Adventures of Comcast Bonnie

I follow @MajorNelson to keep up with the latest Xbox news and cool stuff (not to mention I keep a keen eye out for service issues), and noticed he posted this AWESOME link to the new Xbox Engineering Blog. I read through the newest information about Netflix and Xbox, and noticed a fancypants section about Netflix and streaming.

I am reposting this section with permission from Xbox to shed some light about how this process works, and how Xbox is working to improve the experience for you! Also, I have to throw in the comment about us "throttling" Netflix to keep competition away. This simply isn't the case, and I think the snip from the article below will help clarify what's actually going on. Enjoy!

Special thanks to the Xbox Live Team for putting this up! Remember, the full blog can be reached here!

Streaming Video

Rebuffering
One of the major challenges of streaming video is adapting to changes in available bandwidth. In the first version of Netflix, we saw a number of users experience downgrades in their video quality, accompanied by the “Your Internet connection has slowed” message, shortly after starting playback. This was often due to an ISP boosting available bandwidth for brief periods of time in order to make individual downloads faster. Since we decide which quality level of the content your Internet connection can handle by measuring bandwidth at the very beginning of a download, this boost effect causes the client to believe there is more bandwidth than actually available. Accordingly, we picked a quality level whose bitrate was too high to sustain once the boost ended and the connection reverted to its typical bandwidth characteristics. This resulted in a rebuffering event, otherwise known as a “starvation.”

We’ve done a number of things in this release to reduce the frequency of starvations:

• We have switched to using Netflix’s new, more advanced VC-1 encodings when available. The VC-1 streams are much more efficient than their visually-comparable WMV9 counterparts. This allows us to better utilize available bandwidth by buffering more of the feature in less time.
• We have implemented something we are calling “Seamless Stream Switching.” In the past, a starvation meant having to wait while the app rebuffered a new stream. Now, we have the ability to detect a dip in bandwidth that would cause a starvation, and react before we actually starve. Instead of forcing you to wait while the speed test and buffering is redone, we can prepare the new stream behind the scenes while playing out the old one. See the next section for details on how it all works.
• Courtesy of Seamless Stream Switching, you can now switch up to a higher quality stream. Previously, you would always be locked in to the lower bitrate stream if your bandwidth had degraded during playback. Now, if your bandwidth improves sufficiently, your content bitrate could be upgraded as well.

(Mostly) Seamless Stream Switching
As mentioned above, we can now be proactive about switching to a new stream before a starvation occurs. Here’s some information on how the process works:

Like with the first version of Netflix, a quick speed test is done first to figure out what the Internet connection is capable of. Based on these results, a stream of an appropriate bitrate is chosen - one that should give you the highest video quality possible while still being likely to play back without interruptions (aka starvations).

Unlike the first version of Netflix, the client will continuously monitor the current streaming bandwidth and also the buffer level during playback. The buffer level is how much data has been read ahead from the network connection and buffered up in memory.

If the buffer level falls below a certain threshold–i.e., the buffer level falls into the starvation danger zone–then we start to get interested, algorithmically speaking. At this point, the client looks at several samples of bandwidth measurements to see how the Internet connection is performing, and we’ll also see if buffer levels have been declining. It might seem like a silly thing to do, but the reason we check the latter is that, directly after a seek or after playback starts, buffer levels could technically still be in the red zone; however, as long as they are still increasing, there should be no need for alarm. On the other hand, if buffer levels have been declining, and bandwidth measurements indicate that the client could be in danger of starving, then the client will initiate a switch downwards to a lower bitrate stream. More about that in a moment.

Now if, instead of falling, the buffer level actually rises above a certain threshold, then the client will look at several history samples of bandwidth measurements and see if the Internet connection really can take it to the next level. If so, then the client will initiate a switch up to a higher bitrate stream. Note: To avoid ping-ponging between higher and lower bitrate streams, a backoff period is enforced before switching up if there has been a previous switch down during the playback session. The backoff period increases substantially the more times a downward switch occurs. For example, suppose client starts out playing a piece of content at 2 quality bars, but then bandwidth drops and the client switches down to a lower bitrate stream. At this point, even if the connection improves to the point where the client could technically switch up again, the client will wait an additional period of time before allowing the switch to occur. If the client subsequently performs another switch down during that streaming session, then the backoff time before allowing an up-switch would be longer still. The basic idea is to avoid a cycle where fluctuations in connection bandwidth cause the client to continually vacillate between content quality levels.

Performing the Switch: The Seam
Once a stream switch is kicked off, several things happen. First, the client stops reading any data from the network for the old stream; it plays out whatever buffer has been accumulated for the old stream from here on out. Simultaneously, the newly-freed network resources immediately start grabbing data for the new stream. Once all the data from the old stream has been played, the client will hot-swap to the new stream.

Perhaps this is a good time to confess that it’s not completely accurate to call this process “Seamless Stream Switching.” As it turns out, there is a seam, but it’s very slight and consists of the screen below.

The graphic at the upper right shows you the quality level for the new stream you’ll be watching. You’ll see this screen very briefly (typically, for a second or less) during the switch over from old to new. During this time, the video pipeline is being reinitialized for the new stream. We’ve done as much of the work as we could ahead of time, but there are some things, like initializing the Direct3D device in accordance with the new video resolution, that need to happen right then. This process is fairly snappy but it’s not instantaneous, which is why this small delay exists.

Sometimes the client is unable to adequately buffer up the new stream before the buffer for the old stream runs out. This may happen, for example, if there was a sudden extreme drop in bandwidth or if the Internet connection is not capable of sustaining uninterrupted playback of the lowest bitrate stream. In these cases, you will unfortunately still have to wait to rebuffer while viewing the “Your Internet connection has slowed” starvation screen.

Streaming and Party Mode
The most important thing about the streaming experience in Party Mode is that the video stays in sync among people watching the content. We felt strongly that it would not be a great experience if your buddy laughed at a joke two seconds before you actually saw the funny part of the movie. Because we were optimizing for keeping everybody at the same place in the content, streaming in Party Mode behaves a little differently than streaming in solo mode:

• There is no Mostly Seamless Stream Switching in Party Mode. The seam during the switch, slight as it is, could be enough to throw synchronization off beyond the 500 millisecond threshold that we deemed our maximum allowed drift between participants.
• The headroom that we require for Party Mode playback is higher than for solo mode. For context, headroom is the amount of space required between your measured bandwidth and the selected content bitrate. In other words, for a Party Mode headroom value of say, 30% (hypothetically speaking), your measured bandwidth would need to be at least 1429 Kbps for us to select a 1000 Kbps content stream. This is Bandwidth * (1.0 – Headroom). Now, the reason why we require more headroom in Party Mode is because a starvation in Party Mode affects not only your user experience, but also that of everybody else in your party. When one person starves, everybody else in the party has to wait for that person to rebuffer to keep everyone in sync. We want to make sure that this is very unlikely to occur.

Inside Tips and Tricks for Tweaking Your Netflix Streaming Experience
Since you now know some details of how our streaming system works, there are a couple of things you can do to potentially make the system work better for you.

First off, if you absolutely hate Mostly Seamless Stream Switching, you can prevent us from using it by watching all your content in Party Mode. This works whether you watch by yourself or with others. All you have to do is start a Netflix party. If you choose to do this and don’t want to see your Avatar, then use the On-Screen display and use Display Mode to switch.

Secondly, if you have a spiky or otherwise unstable Internet connection, the Pause button is your best friend. Remember how I mentioned that the client will only consider switching down if 1) the buffer falls below a certain threshold, and 2) the buffer level had been decreasing over the last few seconds? Well, when playback is paused, we will continue to read data from the network, but the buffer can’t decrease since playback has been halted. Therefore, hitting the pause button right after playback starts and leaving the content paused for a short while (30 seconds, a minute or two, see what works best for you) is a cheap and easy way to force the buffer level to increase and potentially avoid a downward switch to a lower bitrate stream.

I had the cool opportunity over the weekend to be a tech for a day. It is something I had been meaning to do for awhile, but never had the time. I was in Cape Cod for the weekend visiting folks, and one of them happens to be a technician for Comcast. I had nothing better to do until they were done working for the day, so I tagged along and finally got some field tech time.


Some of you may have seen the tweets on Saturday. Here's the low-down!

We didn't start the day at this location, which is both a satellite office for the techs and a payment center. Saw the OLD payment center next to it... huge improvement :)



i'm at a customer's house doing a triple play install. of course two gigantic black labs are helping :) 11:39 AM Dec 5th

Having fun with punch blocks... which really aren't that much fun to be honest... 12:34 PM Dec 5th

Did I neglect to mention it was raining? A LOT?



First install finished! Next job involves finding out where a customer's dialtone ran off to. Comcast ninja skillz activate! 1:16 PM Dec 5th

ACTUALLY, we ended up picking up a job that was just a quick install of a cable box at an apartment complex. My tech, however, is a little OCD like me. We ended up cleaning up the entire box for the building so everyone would have happier cable!



Next up: Ooooh triple play install with home networking! Should be tons of fun (no really!) 2:43 PM Dec 5th

I actually spent this job setting up home networking for this customer while Will redid the phone wiring down in the basement. Sometimes, we're not given the most awesome wiring to work with from your old provider, but we'll make it work somehow! :)

(waiting for the test call to confirm service is working)


last job for the day, fixing some phone wiring to restore said rogue dialtone from earlier then done! 5:34 PM Dec 5th

This was actually an interesting dealy-o. The sad part is, we were not able to fix without an electrician (the phone line coming in was just... very unhappy). We got the dialtone, we did not get to fix the underlying problem. The GOOD news is, we know what it is and this customer knows exactly how to fix it. YAY! Also, Mr. Customer was super excited about having a social media person in his house :)

Alas, poor missing dialtone. We knew you well. Everyone should hug their next tech... they probably need it. This isn't an easy job! 5:58 PM Dec 5th

This is so true. It was raining hard, 40 degrees, never know what you're getting yourself into when you go to the next job... every house is different. Every job is a new challenge. I definitely gained a bit more insight into the field tech's day (this was a SLOW day, fyi), and how tough it can be. Gotta say, I slept like a baby that night!

(I was originally going to post this back in September. Oops!)

Since the new school year has begun, let's learn some fun stuff about your internet connection and how to fix it! Prepare to amaze and awe your neighbors, family, and friends who will suddenly deem you as their tech support person :)

OK, class! Let’s begin…

Today, we’ll start off our fun learning series with the simple stuff. When I see problems, the very first thing I'll ask is: Have you reset your modem? If the answer is yes, then I'll ask: Do you have a router? This isn't because I'm trying to blame any particular piece of equipment. You'd be amazed at how many times a router will get "out of sync" with a modem, and vice versa. To fix this…

1. pull the power from both the modem and the router
2. Next, plug the power into the modem ONLY.
3. Wait for the lights on the front to calm down. On Motorola modems, you need four SOLID green lights. (Scientific Atlanta/Webstar needs a SOLID cable light. RCA modems need the Cable Link and Internet light SOLID. If you have a modem not listed here, chances are you bought it or it’s an end of life modem and should be replaced)
4. Once this is done, go ahead and plug the power back into your router and wait for it to finish up it's startup sequence.

Easy enough, right? If you’re still unable to connect, try moving on to the slightly more complicated steps below.

For a direct connection to the modem:

* PC’s: Go to “Network Connections” in your Control Panel. Right mouse click on “Local Area Connection” and click Status. Click the Support tab in this window, and check the IP address.
* Macs: Finder>Applications>System Utilities>Network Utility This will display your current IP address.
* Does the IP start with 169.254? If so, your computer is not getting a connection from the modem and you should probably give us a call. If you see 192.168, you’re connected to a router, have your IP set static, or something else is going on with the modem. Again, something you might want to call us up about.

For connections using a router:

* Make sure your router is obtaining an IP address by going to its home page at http://192.168.1.1 (login required in most cases). You should be able to check on the router's status page and see if you are getting an IP. If you see a 169.254.xxx.xxx address, your router is not connecting to the modem. This could mean that it was not powered on in the right order OR the modem is not assigning IP's.
* Make sure your computer is receiving an IP address from the router by following the steps above. A 192.168.x.x of 10.x.x.x address should show. If not, your router has a problem!
* Make sure you’re connected to your own wireless network. I get more of this than most people think. Be sure to put your wireless network as top priority over others in your network preferences.

For the super geeks out there:

Speaking of modem's not handing out IP's... I see this little problem all the time. The geeks out there seem to notice this the most, and take extraordinary measures to correct the problem. I'll see folks spoofing their MAC Addresses to get the modem to talk to the equipment on the other end. Guess what? You don't need to do that :) ANY TIME you connect something in to your modem, you will need to reset the modem. Otherwise, it will still consider the original device connected, and hence refuse to work with the new device. Easy fix, right? Make sure you’re not overlooking the basics above, since it’s easy to forget the most obvious things sometimes. I once troubleshot a modem for well over an hour, and then discovered it was in standby. The lights on the modem will tell you a lot about what’s going on.

I hope this gives you a little insight into the basics of troubleshooting. Next week, we’ll move on to checking out your internet signal from the comfort of your own home :)

Got a topic I should cover? Let me know in the comments! Got a question about any of the stuff above? Ask away! Let’s keep comments about the lessons on-topic, so we can all chirp in about each thing :)

It's that wonderful time of year again where the leaves change colors, the temperature gets cooler, the cable goes out because of the sun... wait, what?

Yes folks, it's Sun Outage time. It happens a few times a year, and can seriously mess with your cable! Here's the low-down:

From Wikipedia, the free encyclopedia

Here's some info for the techy people out there from the HITS Satellite page:




So, folks... Keep this in mind if you experience blips in your cable network programming!We're not making it up!