Verizon Wireless customers are having trouble accessing the Tripawds community, and VZW tech support fails to acknowledge the network routing issue causing problems. This has been going on for weeks and more than six hours on the phone with call center reps has only resulted in a total customer service failure by Verizon, continued frustration for users, and major stress headaches for me.
Tripawds members need to know that we have been doing our best to get this issue fixed, so I will explain what’s going on and how hard I’ve been trying get help from Verizon network engineers. Besides, such an epic Verizon support failure needs to be shared, if not solely for its entertainment value, then to document what I have discovered in case I ever reach anyone at Verizon who cares, or is able to help.
What Can’t I access My Server via Verizon Wireless Network?
I first noticed issues connecting to the server which hosts the Tripawds blogs and forums when using our Verizon MiFi wireless hotspot on or around March 21, 2015. We were in a remote area at the time, so I presumed that was the reason and continued to work effectively on the Tripawds site by using our HughesNet satellite internet connection. After relocating four times across three states over four days, I was still unable to access our site using the Verizon MiFi, when I had no trouble doing so via HughesNet.
I could use the MiFi to easily browse other websites. Only when attempting to connect to our site, did the Verizon connection timeout. Meanwhile, certain Tripawds members who also happen to be Verizon customers started reporting similar issues.
Thinking there might be a problem with the server, I opened a support ticket with our data center at Peer 1 where we manage our own dedicated server. They were the first to identify serious data loss on the VZW network, prohibiting certain Verizon customers from being able to use our website…
“I did a mtr from your server’s gateway to the destination ip 220.127.116.11 and I am seeing packet loss on verizon’s end (hop 9). If you want us to send an email to Verizon; we could. However please note that since we are not directly peered or their customer, there’s a high possibility that they will not respond back to our email. Please can you ask your client to open a ticket with Verizon regarding the packet lost.” —Peer1 Network Analyst
I’ll be the first to admit that I have a fairly novice understanding of network routing. But let me explain what this means to anyone less technical than I, like the Verizon tech support reps I have been dealing with for instance.
What does mtr mean and what does it do?
The “mtr” stands for My Traceroute. This is a network diagnostics tool run via command line interface, in this case from our server which has a specific IP address on the internet. The results above show nearly 50% packet loss at a hop on the myvzw.com network backbone. Note how the pings (shown at left) never reach me, or my IP address at the time when using my Verizon MiFi.
Compare that to these results from an mtr I personally ran from our server to my IP address when connected via our HughesNet internet connection…
No packet loss, and that last line is my IP address, clearly proving that the connectivity problem is specific to the Verizon network, not a problem with our server. Thus began my journey through tech support hell with Verizon call reps, where the frustration continues.
Before I get into the hilarity that ensued when trying to get help from Verizon support, I’ll share further diagnostic measures we took to confirm this was not an issue with our website, the server, or my MiFi device.
First, I wanted to confirm this same issue was the reason other Tripawds members were also having trouble. So I asked those reporting problems, who also use Verizon Wireless, to provide their IP address…
Surprise! This member in Massachusetts uses Verizon and is unable to connect to Tripawds.
And, this member in Virgina is continually having trouble navigating the forums or posting in the chat room.
So, you may be asking, “Wait Jim, what’s that ‘Level3’ all about? I thought you were blaming this on Verizon.” Well yeah, I am. A supervisor actually mentioned being unable to ping our server from “Level 3” after one of my lengthy support calls, and repeated representative escalations. Well, duh! That’s exactly what I’ve been reporting for the past few weeks now.
Besides, if Level3 is not a Verizon network, they are directly peered to it, and I am a customer of Verizon, not Level 3! But I digress.
Uhh… what’s a ping?
I continued to troubleshoot this issue by “pinging” our server from both networks I have available for accessing the web. The “ping” is a simple shell command often used for network diagnostics to determine if a certain point on the internet can be reached from your connection. In our case, I pinged the Tripawds server from both our Verizon MiFi and HughesNet modem. Here’s what I found…
Notice how I have no problem communicating with our server via our HughesNet connection. That is our server quickly responding to confirm each ping was received. When attempting to do the same via our Verizon MiFi, each ping request merely times out. Sigh…
At this point, over a week ago, I had consulted with our data center, my server manager, and a number of internet experts and very technically adept friends. We all agreed that the problem lies in the network, there is no problem with my server or website configuration. And my MiFi is working fine.
So, about that “entertainment factor” I referred to earlier…
Does anyone at Verizon Support Understand Networking?
This support debacle is not just funny, it is pathetic. It might be amusing, if it wasn’t my server, and my customers who are are being impacted by the problem. I just can’t believe that I know more than every Verizon tech support supervisor I have spoken with. The following are just a few highlights from the hours I have spent escalating calls only to have two support tickets closed with unacceptable resolution.
This is a text message I received from Verizon after a supervisor told me the issue had been “resolved” followed by a screenshot of what I had spent more than an hour explaining to various reps. Why a text? Yeah, that’s funny in itself. A rep at the Verizon call center in Idaho told me they do not have email. Yes, an internet support center that cannot email. But wait, it gets funnier. The rep also told me they do not even use Verizon services at their location!
Frustrated, I had to call back and find out how they could possibly come up with this “resolution” for the issue I’ve reported and explained in detail multiple times. Get this: after another couple hours speaking with various reps, and their supervisors, Verizon told me “your IP address is bad.”
Funny Thing 1: Verizon Support closes both of my tickets, after determining my IP address is invalid by entering the address into a web browser and getting a page that says website unavailable. 😕
Anyone with a basic understanding of Internet Protocol or server management knows that this page actually proves the IP address is valid and the server is doing its job. We have multiple websites and 1,000+ subdomains configured at this IP address. How would a server know which one the user wants to visit? Here is further proof that our server is configured properly, with multiple domains at the IP address.
Enough said. Back to Verizon’s supposed “resolution” to our problem…
Funny Thing 2: Not being able to ping the IP address via the Level 3 network? That is exactly my point!
Funny Thing 3: I was never notified that either of my support request tickets had been “resolved” even after speaking with one supervisor who actually sounded like she may be able to help by discussing the problems with engineering. She promised that I would receive a text message allowing me to communicate with her directly, bypassing the multiple timely escalation requests every time I called.
She downright lied, or doesn’t even know how her own support system works, because when I replied the message, it came right back to me.
Funny Thing 4: I call back to express my concerns once more, and explain the problems all over again. I press the option for “tech support” so I can speak with a human. Seriously, after 20+ minutes on the phone with a rep who clearly had no clue what I was talking about, I am told, “OK, let me connect you with technical support.”
“I thought this was tech support!” I say
“No, this is customer service,” she says. Service? Not!
Various Funny Things: I did my best to remain calm, but after a few calls, I knew the drill. I would begin each call by screening the support rep to see if they know what I was talking about.
After trying to explain to one rep that this was not a problem with my device, but a network backbone routing issue, I was transferred and told…
“Please hold, we’ll be happy to help you get your backphone working.”
Another time, after clearly describing the packet loss at a hop on the Verizon network, the rep said
“I’m sorry to hear about your package loss, let me transfer you to someone who can help with that.”
Still, no help. So I turned to social media…
Apparently Verizon handles calm requests for help on Facebook by deleting posts to their page, without reply.
On Twitter @VerizonSupport was actually quite responsive with a reply suggesting I follow and send a direct message. We replied back and forth via private messages immediately. Great! or so I thought…Sigh…hoping @VZWSupport might be as responsive, I send various messages and screenshots. It’s not easy describing such technical issues in 140 characters or less, but I was polite and clear providing as much detail as possible. After waiting almost 48 hours, I get this reply to my message with a screenshot clearly showing network data loss…
Seriously? Security setting? On who’s device? This is not just my problem! It is with Verizon users all over the country. I’ve asked them to clarify “security setting” since there are no IP blocks on our server for Verizon users. Could Verizon be blocking the IP on their end? I have yet to hear a reply.
Another Funny Thing: The link they sent me is for instructions about how to use my MiFi and reset its admin password.
So here we are, weeks later with nobody willing or able to help.
Where is my Support Hero?
Please, somebody. Tell me I’m wrong. I would love for someone to tell me what’s going on and how I can fix it. That person clearly doesn’t work in technical support at Verizon. I have been told, “reporting this issue again will not help” and that I need to contact my web host. They even kindly told me who that was, as if I didn’t know!
If you know anybody at Verizon who may be able to help, please forward the link to this post!
If you can help, please leave a comment with any suggestions.
If you agree with anything I’ve said, or have had similar experiences dealing with Verizon please share this link.
And if you are a Tripawds member, please know that we’re trying desperately to et this issue resolved. Sadly, if you’re experiencing this issue, you likely won’t be able to read this. 🙁
UPDATE: April 20, 2015
Big surprise. Verizon Wireless has now closed the third support ticket about this issue, insisting that is is still “a problem with the site.”
Furthermore, we have received confirmation from network analysts that the packet loss we have identified at VZW network hops is being caused by ICMP throttling at the network nodes.
ICMP Rate Limiting is a common practice to alleviate network congestion by prioritizing “more important” traffic. How do I explain to Tripawds users that their beloved pets are not important enough for anyone at VZW support to help?
UPDATE: May 10, 2015
Finally. After getting nowhere with Verizon Wireless technical “support” for nearly two months I believe this problem has actually been resolved! At least so far…After I escalated the issue to Executive Management, I was able to work directly with network techs who actually understood me, and identified a problem with packet optimization that wasn’t working properly for everybody. Persistence pays off!
9 thoughts on “Verizon Wireless Technical Support Fails to Help Tripawds Users”
HOOOOLLLLLYYYY COW!!! Okay….let’s get real…HOOOOOLLLLYYY SH*T!!!!!
The fact that you haven’t completely gone off the deep end and are still able to communicate is a testament to your character and your ability to keep insanity from taking over!!!
Do I understand what you’re talking about? Actually…and this is shocking….a teeny weeny tiny bit…kinda…sorta…..in a conceptual way. Can I relate to frustration with Verizon???? YES!!! The thing is, they didn’t have a computer dummy on the other end of the line when they were talking to you, and yet you still got similar treatment! I would think I was being transferred to a Level-whatever -the -heck- they -are -Tech, only to find out itnwould be another customer service rep!
And for the love of goD….PACKAGE LOSS????? PA C K A G E. L O S S????? I think Rene would no long before Verizon if you lost your package!!!
I did a search on my “apple computer” to see if I could find results for VERIZON CUSTOMER SERVICE AND TECHNICAL SUPPORT…
This is what I found…
I really do hope that though this blog and the vast network we have here, someone can make the right connection for us. I’m trying to get an appt. with my BFF Bobby at Verizon on something else and I will DEFINITELY take this blog with me. Bobby cares!
OH! WAIT!! My bad….that would be a waste of time because they told us it wouldn’t do any good to report it again!! Now THAT’S called customer “no support” service!!
You’ve got the whole community behind you and together we will get results!!!
With huge amounts of gratitude for what you do to keep this INVALUABLE site up and running..
Sally and Happy Hannah and Merry Myrtle too!
Heck…my picture of my apple computer results didn’t show up…trying again..
OMD. WTF. I am so sorry you are having to go through this. We knew from experience that Verizon tech support sucks but this is beyond the beyond. Here’s hoping you have a good supply of “Mothers Little Helper” to get you through this!
Sending you lots of peaceful vibes from the OP
Holy Crap, my phone stopped working one day. I would make a call and then it would shut off. I called Verizon and because they couldn’t call my phone and verify it was mine they said I would have to drive to LA and show them my phone. I beat my phone against my countertop and broke it in a zillion pieces. I would’ve destroyed a lot of stuff dealing with what you are dealing with. Not pretty but hey I felt better.
OH HOLY CRAP… this is horrible 🙁
I am sorry you are dealing with this. I would have screamed and pulled out my hair by now
Okay…so I go visit my BFF at Verizon today…….
While trying to explain where to find my document download (yeah, this is the kind of stuff he has to xeal with. But I always buy something to ease my guilt), I ask if he’ll take a look at your post.
WELL! BFF Bobby at Verizon got a little twinkle in his eye as he pushed my mundane stuff aside and started studying your post!! He saw a challenge and he was excited!!
BFF Bobby at Verizon started having a quizzical look across his face and his brow became furowed. I knew this was not good. He continued to try though…he would not back away.
Weeeelll….he finally emailed it to himself and he’s going to work on it some more later and find the right people to help!
AND…he’s determined…DETERMINED..to find help!!
BFF BOBBY at Verizon CARES!!!
This time I brought a tavlet cover with all sorts of “positions” from him. I coukd provably get it for half the cost at Amazon! He put it on for me.
Now, however, I have to go back to BFF Bobby at Verizon. I want to take it off…can’t get it off…afraid I’ll break it. I know he just “snapped” it in place, but it’s like it’s glued on! Geez,and you think you have problems!
I read through this article. Your issue is with a level3 router 8-1-7.ear1.dallas1.level3.net based on the screenshot. A whois for level3.net shows the following tech support number.
I’ve needed to work with other large companies with similar routing issues. It isnt as hard as it would seem, once you figure out how to contact them.
Hope it helps if this is still ongoing.
Thanks Tim, it ended up not being a Level3 issue after all. Which still baffles me considering the packet loss shown at their Dallas hop. But after escalating this issue to our Regional President, I worked directly with the VzW Technical Advisory Group to identify a problem with data optimization they were performing that adversely impacted certain traffic.