After getting absolutely nowhere with Verizon Wireless technical âsupportâ for two months, spending hours on the phone with reps who knew less about Internet Protocol than I doâwhich is pretty patheticâand tearing half my hair out, I believe the problem with Verizon Wireless users having difficulty accessing the Tripawds Blogs and Forums has finally been resolved, once and for all. At least so far…
We’ve been using our Verizon Mifi to successfully manage Tripawds and the other websites we host for about a week now, which has been impossible to do for the last couple months due to packet loss I identified using traceroutes from our server. Don’t worry, I’m not going to get all technical again. Pardon my French, but this shit is above my pay grade, and the debacle deserves at least a little cussing.
Never mind what I feel like telling all those “techs” who insisted it was a “problem with our site,” or that our “IP address is bad.”
Instead, I’d prefer to put this behind us, and hop on. Thank you Jerry for teaching me to Be More Dog.
What happened and how did we fix it?
After making a stink on social media, I thought I was finally getting the attention of somepawdy at Verizon who could help. Apparently not.
Only after escalating the issue to Executive Management did I speak with someone who, “sincerely wanted to help get this fixed.” I discovered how to contact your Regional Verizon Wireless President, and didn’t expect to hear back from him directly, if at all. So I didn’t get my hopes when the woman I spoke with acted very diplomatic and promised to put me in touch with a network engineer. The next day, however, I was elated after receiving a call from a network tech at their Technical Advisory Group who actually admitted there was clearly a problem and promised to get to the bottom of it.
To make a long story short, I spent the next week on the phone working directly with a number of technicians who worked their magic while running live trace reports on my device while connected to our server. They identified a problem with packet/data optimization Verizon performs that was not working properly for certain network traffic. Lucky us!
They also apologized for the prior lack of support I had received, and we had a good laugh at some of the responses I was given during my many previous pleas for help.
This whole ordeal would be funny, if it wasn’t our sites, and Tripawds users who were being affected. If you use a Verizon Wireless device to access Tripawdsâand you can read thisâplease let us know if you’re still having any trouble. If you know any other members who may have given up in frustration, please tell them to try again.
And if you appreciate the efforts and persistence that went into getting this resolved, please consider Supporting Tripawds! Had I given up, I’m certain the problem would remain today.
Verizon Wireless customers are having trouble accessing the Tripawds community, and VZW tech support fails to acknowledge the network routing issue causing problems. This has been going on for weeks and more than six hours on the phone with call center reps has only resulted in a total customer service failure by Verizon, continued frustration for users, and major stress headaches for me.
Tripawds members need to know that we have been doing our best to get this issue fixed, so I will explain what’s going on and how hard I’ve been trying get help from Verizon network engineers. Besides, such an epic Verizon support failure needs to be shared, if not solely for its entertainment value, then to document what I have discovered in case I ever reach anyone at Verizon who cares, or is able to help.
What Can’t I access My Server via Verizon Wireless Network?
I first noticed issues connecting to the server which hosts the Tripawds blogs and forums when using our Verizon MiFi wireless hotspot on or around March 21, 2015. We were in a remote area at the time, so I presumed that was the reason and continued to work effectively on the Tripawds site by using our HughesNet satellite internet connection. After relocating four times across three states over four days, I was still unable to access our site using the Verizon MiFi, when I had no trouble doing so via HughesNet.
I could use the MiFi to easily browse other websites. Only when attempting to connect to our site, did the Verizon connection timeout. Meanwhile, certain Tripawds members who also happen to be Verizon customers started reporting similar issues.
Thinking there might be a problem with the server, I opened a support ticket with our data center at Peer 1 where we manage our own dedicated server. They were the first to identify serious data loss on the VZW network, prohibiting certain Verizon customers from being able to use our website…
“I did a mtr from your server’s gateway to the destination ip 70.196.205.9 and I am seeing packet loss on verizon’s end (hop 9). If you want us to send an email to Verizon; we could. However please note that since we are not directly peered or their customer, there’s a high possibility that they will not respond back to our email. Please can you ask your client to open a ticket with Verizon regarding the packet lost.” âPeer1 Network Analyst
My Traceroute Results from Server to Verizon MiFi
I’ll be the first to admit that I have a fairly novice understanding of network routing. But let me explain what this means to anyone less technical than I, like the Verizon tech support reps I have been dealing with for instance.
What does mtr mean and what does it do?
The “mtr” stands for My Traceroute. This is a network diagnostics tool run via command line interface, in this case from our server which has a specific IP address on the internet. The results above show nearly 50% packet loss at a hop on the myvzw.com network backbone. Note how the pings (shown at left) never reach me, or my IP address at the time when using my Verizon MiFi.
Compare that to these results from an mtr I personally ran from our server to my IP address when connected via our HughesNet internet connection…
MTR results from server to my HughesNet ip address
No packet loss, and that last line is my IP address, clearly proving that the connectivity problem is specific to the Verizon network, not a problem with our server. Thus began my journey through tech support hell with Verizon call reps, where the frustration continues.
Before I get into the hilarity that ensued when trying to get help from Verizon support, I’ll share further diagnostic measures we took to confirm this was not an issue with our website, the server, or my MiFi device.
First, I wanted to confirm this same issue was the reason other Tripawds members were also having trouble. So I asked those reporting problems, who also use Verizon Wireless, to provide their IP address…
mtr results from server to Massachusetts Verizon IP address
Surprise! This member in Massachusetts uses Verizon and is unable to connect to Tripawds.
Data loss on Level 3 network for VA Verizon user.
And, this member in Virgina is continually having trouble navigating the forums or posting in the chat room.
So, you may be asking, âWait Jim, what’s that âLevel3â all about? I thought you were blaming this on Verizon.â Well yeah, I am. A supervisor actually mentioned being unable to ping our server from âLevel 3â after one of my lengthy support calls, and repeated representative escalations. Well, duh! That’s exactly what I’ve been reporting for the past few weeks now.
Besides, if Level3 is not a Verizon network, they are directly peered to it, and I am a customer of Verizon, not Level 3! But I digress.
Uhh… what’s a ping?
I continued to troubleshoot this issue by “pinging” our server from both networks I have available for accessing the web. The “ping” is a simple shell command often used for network diagnostics to determine if a certain point on the internet can be reached from your connection. In our case, I pinged the Tripawds server from both our Verizon MiFi and HughesNet modem. Here’s what I found…
Comparison of server ping results via different networks.
Notice how I have no problem communicating with our server via our HughesNet connection. That is our server quickly responding to confirm each ping was received. When attempting to do the same via our Verizon MiFi, each ping request merely times out. Sigh…
At this point, over a week ago, I had consulted with our data center, my server manager, and a number of internet experts and very technically adept friends. We all agreed that the problem lies in the network, there is no problem with my server or website configuration. And my MiFi is working fine.
So, about that “entertainment factor” I referred to earlier…
Does anyone at Verizon Support Understand Networking?
This support debacle is not just funny, it is pathetic. It might be amusing, if it wasn’t my server, and my customers who are are being impacted by the problem. I just can’t believe that I know more than every Verizon tech support supervisor I have spoken with. The following are just a few highlights from the hours I have spent escalating calls only to have two support tickets closed with unacceptable resolution.
This is a text message I received from Verizon after a supervisor told me the issue had been “resolved” followed by a screenshot of what I had spent more than an hour explaining to various reps. Why a text? Yeah, that’s funny in itself. A rep at the Verizon call center in Idaho told me they do not have email. Yes, an internet support center that cannot email. But wait, it gets funnier. The rep also told me they do not even use Verizon services at their location!
Frustrated, I had to call back and find out how they could possibly come up with this “resolution” for the issue I’ve reported and explained in detail multiple times. Get this: after another couple hours speaking with various reps, and their supervisors, Verizon told me “your IP address is bad.”
Funny Thing 1: Verizon Support closes both of my tickets, after determining my IP address is invalid by entering the address into a web browser and getting a page that says website unavailable. đ
Anyone with a basic understanding of Internet Protocol or server management knows that this page actually proves the IP address is valid and the server is doing its job. We have multiple websites and 1,000+ subdomains configured at this IP address. How would a server know which one the user wants to visit? Here is further proof that our server is configured properly, with multiple domains at the IP address.
Enough said. Back to Verizon’s supposed “resolution” to our problem…
Â
Funny Thing 2: Not being able to ping the IP address via the Level 3 network? That is exactly my point!
Funny Thing 3: I was never notified that either of my support request tickets had been “resolved” even after speaking with one supervisor who actually sounded like she may be able to help by discussing the problems with engineering. She promised that I would receive a text message allowing me to communicate with her directly, bypassing the multiple timely escalation requests every time I called.
She downright lied, or doesn’t even know how her own support system works, because when I replied the message, it came right back to me.
Funny Thing 4: I call back to express my concerns once more, and explain the problems all over again. I press the option for “tech support” so I can speak with a human. Seriously, after 20+ minutes on the phone with a rep who clearly had no clue what I was talking about, I am told, “OK, let me connect you with technical support.”
“I thought this was tech support!” I say
“No, this is customer service,” she says. Service? Not!
Various Funny Things: I did my best to remain calm, but after a few calls, I knew the drill. I would begin each call by screening the support rep to see if they know what I was talking about.
After trying to explain to one rep that this was not a problem with my device, but a network backbone routing issue, I was transferred and told…
“Please hold, we’ll be happy to help you get your backphone working.”
Another time, after clearly describing the packet loss at a hop on the Verizon network, the rep said
“I’m sorry to hear about your package loss, let me transfer you to someone who can help with that.”
Still, no help. So I turned to social media…
Apparently Verizon handles calm requests for help on Facebook by deleting posts to their page, without reply.
On Twitter @VerizonSupport was actually quite responsive with a reply suggesting I follow and send a direct message. We replied back and forth via private messages immediately. Great! or so I thought…Sigh…hoping @VZWSupport might be as responsive, I send various messages and screenshots. It’s not easy describing such technical issues in 140 characters or less, but I was polite and clear providing as much detail as possible. After waiting almost 48 hours, I get this reply to my message with a screenshot clearly showing network data loss…
Seriously? Security setting? On who’s device? This is not just my problem! It is with Verizon users all over the country. I’ve asked them to clarify “security setting” since there are no IP blocks on our server for Verizon users. Could Verizon be blocking the IP on their end? I have yet to hear a reply.
Another Funny Thing: The link they sent me is for instructions about how to use my MiFi and reset its admin password.
So here we are, weeks later with nobody willing or able to help.
Where is my Support Hero?
Please, somebody. Tell me I’m wrong. I would love for someone to tell me what’s going on and how I can fix it. That person clearly doesn’t work in technical support at Verizon. I have been told, “reporting this issue again will not help” and that I need to contact my web host. They even kindly told me who that was, as if I didn’t know!
If you know anybody at Verizon who may be able to help, please forward the link to this post!
If you can help, please leave a comment with any suggestions.
If you agree with anything I’ve said, or have had similar experiences dealing with Verizon please share this link.
And if you are a Tripawds member, please know that we’re trying desperately to et this issue resolved. Sadly, if you’re experiencing this issue, you likely won’t be able to read this. đ
UPDATE: April 20, 2015
Big surprise. Verizon Wireless has now closed the third support ticket about this issue, insisting that is is still “a problem with the site.”
Furthermore, we have received confirmation from network analysts that the packet loss we have identified at VZW network hops is being caused by ICMPÂ throttling at the network nodes.
ICMP Rate Limiting is a common practice to alleviate network congestion by prioritizing “more important” traffic. How do I explain to Tripawds users that their beloved pets are not important enough for anyone at VZW support to help?
UPDATE: May 10, 2015
Finally. After getting nowhere with Verizon Wireless technical “support” for nearly two months I believe this problem has actually been resolved! At least so far…After I escalated the issue to Executive Management, I was able to work directly with network techs who actually understood me, and identified a problem with packet optimization that wasn’t working properly for everybody. Persistence pays off!
The fight against Spam Blog registrations for any WordPress MU site administrator â that’s me â is an ongoing battle, that often seems to have no end. But at the risk of jinxing things I would like to think we are finally winning the war against these splogs, which are much worse than any old Spam.
Tripawds members can feel free to report questionable blogs here in the Tech Support discussion forum. Just keep in mind, that said blogs may very well be deleted by the time you do so since we are always doing our best to immediately get rid of any that actually get created. We just can’t do so in our sleep. Thank Dog there are tactics that can!
WPMU site administrators may be interested in the available anti-splog options and these steps we took to slow the flow of splogs here at Tripawds …
Restrict WPMU User and Blog Registration
The only bulletproof way to keep splogs out of your WPMU site is to disable registration, and require users to email requests for signup. We’re not about to do that since we want all new users to have immediate access to our canine cancer discussion forums, and be able to start sharing their three legged dog stories right away.
The Secure Invitations WPMU plugin stops access to your signup page, except where the visitor has been invited and clicked the link in their invitation email. This seems like a great solution for stopping automated splog signups on private or corporate installations, but it would not work for us. Too many people find Tripawds when searching for help to allow registration on an invitation only basis.
Edit .htaccess to Stop Automated Spam Blogs
One sure fire way to stop automated spam blog registrations is to ban any signups resulting from POST requests (form submissions) for wp-signup.php that have not been sent from a web page within the WPMU site itself. D’arcy Norman makes this easier than it sounds.
For complete details, or to ask any related questions, please visit his original post about how to stop WPMU spam blogs with .htaccess file edit. Be sure to read the comments there as many concerns have been addressed.
To summarize, however, adding the following to your .htaccess file will indeed stop most if not all automated spam blog registrations:
# BEGIN ANTISPAMBLOG REGISTRATION
RewriteEngine On
RewriteCond %{REQUEST_METHOD} POST
RewriteCond %{REQUEST_URI} .wp-signup\.php*
RewriteCond %{HTTP_REFERER} !.*yourdomain.tld.* [OR]
RewriteCond %{HTTP_USER_AGENT} ^$
RewriteRule (.*) http://die-spammers.com/ [R=301,L]
# END ANTISPAMBLOG REGISTRATION
All you need to do is change yourdomain.tld to the domain of your WPMU website, and change die-spammers.com to wherever you want potential spammers redirected. We send them here for an explanation, since there may be a rare occasion when legitimate users might get blocked â if they clicked the signup link from an email they were sent, for instance.
Another more serious tactic to deploy is banning any spam blog registrations from specific ip addresses. This can also be done by editing your .htaccess file. Check your server logs or new registration emails to find out where theyâre coming from and then block their IP addresses in your .htaccess file by adding the following:
# BEGIN IP ADDRESS BAN
order allow,deny
deny from 192.168.44.201
deny from 224.39.163.12
deny from 172.16.7.92
allow from all
# END IP ADDRESS BAN
Or, block a range of IP addresses:
# BEGIN BAN IP ADDRESS RANGE
order allow,deny
deny from 192.168.
deny from 10.0.0.
allow from all
# END BAN IP ADDRESS RANGE
Please note that the IP addresses shown are examples only! Special thanks to Sarah over at WPMU.org for this one, and other tips to get rid of spam blogs once and for all.
Stop Common Spam Users in Their Tracks
Using cPanel …
At this point, I thought we had defeated the sploggers. But alas, before long we had more Money Making Tips for Tripawds. After getting tired of editing our .htaccess file, I took to banning IP addresses directly from cPanel using the Quick Deny feature of csf – ConfigServer Firewall. This is much quicker, and enables me to easily search for banned IP addresses should a legitimate user get blocked, which has not happened yet.
Banning known spammers …
Some of the most common spam blog registrations come from easily identifiable sploggers. One such known malicious username format is fullname#### â such as geraldmason1976. No such users will ever register for Tripawds Blogs ever again thanks to our WPMU Dev Premium membership. With our membership, I am able to particiapte in the WPMU Dev Premium Support Forums where I helped test this Splog Check for Known WPMU Spammers…
Within the wpmu_validate_user_signup function, I added the following hack to disallow any username ending in four digits:
// BEGIN 4-DIGIT USERNAME BAN
$tmp_user_name = substr($user_name, -4);
if (is_numeric($tmp_user_name)) {
$errors->add('user_name', __("Username not allowed"));
}
// END 4-DIGIT USERNAME BAN
Special thanks to Dr. Mike for this one. For detailed instructions, and to take advantage of many other benefits including Premium WPMU plugins and themes, please sign up for a WPMU Dev Premium membership. And if anyone is wondering why they couldn’t join Tripawds with a username ending in four digits, that’s why!
But it still wasn’t enough to keep me from deleting far too many spam blogs every morning. It was time to edit some core files…
Reword Signup Page to Confuse Sploggers
Many sploggers use Google to find WPMU signup pages by searching for their default content. Once they find pages including phrases like “Gimme a blog!” or “Just a username, please.” they know where to go to create their next blog for stretch mark creams or making money online.
By rewording the content within our wp-signup.php file, we were able to thwart some more spam blog registrations. Here is the default content from the lines I edited to re-phrase:
#157:'Get <em>another</em> %s blog in seconds'
#178:"If you’re not going to use a great blog domain, leave it for a new user. Now have at it!"
#241:'Get your own %s account in seconds'
#254:'Gimme a blog!'
#257:'Just a username, please.'
Be sure to use proper HTML entities where necessary, and do not edit any PHP code â like %s whatever that means. Keep a copy of the original file to be safe! And for the record, this information is based on WPMU 2.8.6 â line numbers and code may or may not change in future revisions!
This alone was still not quite enough to keep the sploggers at bay. More extreme measures measures were necessary…
Rename WPMU Signup Page to Stop Splogs
Savvy sploggers know to search for the signup page by its file name. A simple Google search like âGimme a blog!â inurl:wp-signup.php will result in numerous sites to attack, for instance. Nefarious automated spam bots also know to go directly to wp-signup.php for immediate registration. Renaming the WPMU signup page was the final step we took to slow splog registrations to a crawl here at the Tripawds Blogs community.
There are a few important steps, however, to keep in mind when taking this measure to stop spam blogs.
If you rename wp-signup.php there are five core file locations where the link must be updated:
wp-signup.php: rename file and update link to match on lines #26, #179, #242, #311, & #391
wp-login.php: update link to match on line #390
wpmu-settings.php: update link to match on line #154
Again, these are the line numbers in WPMU 2.8.6 â future revisions might differ. Now, the reason I put off doing this for so long was because we had linked directly to our signup page in numerous blog posts and discussion forum topics. Had I renamed the signup page to anything other than wp-signup.php all these existing links would be broken.
So, after discovering the following SQL query to easily search and replace terms via PHP MyAdmin, I went for it:
UPDATE wp_{id}_posts SET post_content = REPLACE (post_content, 'wp-signup.php', 'wp-renamed.php')
Simply replace {id} with the ID number of the blog you want to search. And replace ‘wp-renamed.php’ with whatever you renamed your signup page. To run the same search and replace query within SimplePress Forum posts, just replace wp_{id}_posts with wp_{id}_sfposts. Also, be sure to edit any theme files or sidebar widgets you may have that point directly to your signup page.
Recommended solution for stopping spam blogs
The single best suggestion I have to stop all WPMU splog registrations forever is: all of the above. And I recommend implementing these steps immediately after installation of WordPress MU â before you enable registrations.
Whatever you do, don’t give up and disable registration. Do that and the sploggers win!
