Here's a set of questions back from the Extender team:
Can you ask the user for info on which threads he/she upped the priority of that resulted in things working?
Also, additional info regarding the encoding parameters of the WMVs in question would be very helpful (video bit-rate, CBR vs. VBR video, audio type (WMA Pro/Standard/Lossless), audio bit-rate, and CBR vs. VBR audio).
Hey Jessica,
Thanks for the response. I tried a private message to the person who posted about thread priorities but haven't got a response - the post was several weeks ago so I might not hear from person at all. It was in the xbox forum that I referenced in my first email.
I can definitely give you my encoding parms. I only use the Windows Media Encoder software that has been out for several years and I encode my home video to one of two custom profiles that I use to archive stuff.
The first is Windows Media Video 9, VBR Quality 98 average bit rate 10mbps - audio is Windows media audio 9.2 VBR Quality 98 44kHz. This is a two pass mode.
The second is Windows Media Video 9, VBR quality 99 - audio is the same as above. This is a one pass mode that usually gives bit rates in the 16-20 mbps range.
I realize that the bit rates are fairly high but I have been doing this for a couple years and have a few hundred gigabytes of these files now and I have not had a single glitch in streaming these files to the 360 from MCE 2005. This is after watching dozens and dozens of hours of video.
After upgarding to Vista every single file has a 2-3 second freeze every 5 - 10 minutes, often times in the same location but not always. This is why I'm confident that it is Vista related. That and reading that several other folks have the same problem after upgrading to Vista.
I'm very motivated to solve this problem since I've already built such a huge library of this stuff. I would be happy to send you the actual profiles I use (.prx files) if you private messaged me an email address. I don't know how to include attachments here. I'll even send a DVD with some video files on it if that would be helpful.
Thanks again for the help!
Okay, I decided to play around with task manager and the “Reliability and Performance monitor” to see what I could find and I think I may have isolated the problem. Jessica, this post will be kind of long I expect, but if you could forward it to the extender team I think we may be a lot closer to fixing it. As “mpickup” suggested I’m pretty sure the problem is in ehshell.exe (someplace in those ~34 threads!) but I think it is more of a CPU problem that a disc I/O problem.
I started with a single 12 minute video clip encoded at 2 pass 10mbit/sec average VBR with windows media encoder (I mentioned this profile in the previous post). This was originally DV camcorder video so it is 720x480. This clip has a “glitch” at precisely 8 minutes 15 seconds every single time it is played through the 360 MCE interface. I decided to play this clip 3 different ways and watch the CPU usage and I/O rates coming off disk and going over the network. The first way was locally on the Vista machine through Windows Media Player. The second way was streamed to the 360 through the dashboard media connect interface. Lastly, of course, was the problematic streaming to the 360 through MCE.
In all three cases the I/O rates off disc looked appropriate averaging ~10mbit/sec over the whole clip but rising to nearly 16 mbit/sec for around 60 seconds in the area of the glitch (the glitch is roughly centered in this area). There is fast motion in this area (kids on a merry-go-round) so that’s expected with VBR. The two streaming cases saw the same data rate across the network. Ehshell.exe seems to be doing 10-50 disc IOs/sec during the MCE case – that may be just fine I expect. The problem is with CPU usage:
2.4 Ghz Core 2 Duo (Intel E6600) - Vista Ultimate (32 bit)
Avg. CPU % CPU % near 3 second glitch/freeze area
Local WMP playback 10% 13% (mostly in mfpmp.exe)
360 stream from dashboard 4% 5% (mostly in wmpnetwk.exe)
360 stream from Vista MCE 20% 40% (all in ehshell.exe)
Check out those numbers. It takes several times the CPU power to stream that data to the 360 MCE extender as it does to decode and play it back locally! That’s wacky. The CPU numbers to stream to the 360 dashboard seem fairly reasonable and there is never any glitch with this path (or local playback of course). Things then get even stranger.
After I hit the glitch point at 8:15 into the video the CPU number slowly drops until around 9 minutes it is now down to < 2% yet the data rate is still at or above 10 mbits/sec. The CPU stays at 2% until the video ends. Now at the MCE interface on the 360 I have the 3 options DONE, RESTART, or DELETE. If I choose RESTART right there the video replays again streaming the same data rate off disc and across the network as before but the CPU usage in ehshell.exe stays at 2% for the whole video – needless to say there is no glitch. But if I do *anything* else in the MCE U/I like click DONE and then restart the video, then it’s back to 20-40% CPU usage in ehshell.exe and the glitch returns. Wacky huh? The 2% CPU number sounds about right for simply grabbing 1-2mbytes/sec data off my SATA-2 disc and shunting it to the gigabit network port – this should take very little CPU. And it looks like in certain cases ehshell.exe is capable of doing this. But usually it takes 10+ times as much CPU to accomplish the same task. I did try changing the priority of ehshell.exe from “above normal” to “high” but this didn’t fix the problem.
If someone figures out where that mystery CPU power is being wasted in ehshell.exe, I’ll bet you’ll have solved this problem for everybody.
Jessica, please let us know what the extender team thinks of this and let me know if I can provide any more details.
Thanks again!
WOW. Thank you very much for your detailed information!
Folks on the Extender team have read this thread and will investigate. I can't promise when we'll have a solution, but the problem is definitely in the right hands now.
JessZahn: WOW. Thank you very much for your detailed information! Folks on the Extender team have read this thread and will investigate. I can't promise when we'll have a solution, but the problem is definitely in the right hands now.
Jess,
Its good to see you have a presence on this website. The best testing lab in the world can't test every possibility. A fairly well organized user group of early adopers like this is just what you need to work out the rest of the kinks before the mainstream starts stumbling on them.
Here is a detailed list of test cases which, if the right engineer looks at it, should serve to narrow the problem down quickly:
http://thegreenbutton.com/forums/permalink/174674/170628/ShowThread.aspx#170628
It is clear the Xbox360 Media Extender playback mechanism is throttling the network traffic during playback of these .WMVs to about 7Mbps by triggering something in pacer.sys on the PC. This appears to be due to a module that changed since MCE2005 and uses more resurces generating a shortage that creates a buffer overrun. When the buffer is full, flow control happens (I am guessing session layer), the video pauses for four seconds, but resumes where it left off. When pacer.sys is stopped on the PC, bandwidth is not throttled, video pauses, but overrun data is lost...the video pauses for 30 or more seconds and resumes 30 or more seconds later in the film. Re-enabling pacer.sys resumes 4-second pause behavior like flipping a switch. It makes it pretty clear the resources shortage is downstream from there in the playback mechanism.
To further narrow this down you can observe that:
If you need a copy of the .wmv file, send me an ftp site and I will upload the file to you. Email is phil@mccalley.net.
Hi All,
I have the problem as described but can also force the same symptom by rewinding live TV or recorded TV.
I ran similar tests although I compared performance of the Media Centre PC when playing the video via XBOX 360 against playing the same video on the PC itself.
If I watch Live TV on the 360 and fast rewind (about a minute back), the ehshell CPU on the PC goes to around 40% as reported above and stays there. It takes maybe 5 secs for the picture to start breaking up. If I fast forward to real time, the ehshell cpu drops immediately back to normal (2-4%).
If I do the same thing on the Media PC the CPU never struggles, no matter how hard I try. Even more puzzling is that`even when the CPU is running at 60% because of the XBOX testing, live and recorded TV on the PC is absolutely fine. Which suggests that the CPU is not actually causing the problem on the XBOX.
At one point I suspected that the problem is related to the Gigabit LAN and Vista. There are reports on the Microsoft site about Gigabit switches etc not handling video loads properly. I cant find any evidence to support this. I am using an Intel DG965WHMKR which has onboard 82566DC Gigabit Lan - which someone earlier in the thread also has. Do others with this problem have the same board/Lan chipset?
Coma.
I have solved my problem - I plugged in a Gigabit PCI card and disabled the onboard LAN. I am using an Intel DG965WHMKR board which has the 82566DC gigabit LAN.
CPU now stays at 1-2% even when XBOX is playing recording etc.
Maybe I didnt have exactly the same problem as reported by other people but I think the fact that my symptoms of heavy CPU usage (50%) for ehshell suggests the problem is related.
Good luck.
Coma - Thanks for the update.
Jess - Don't drop the ball. The LAN card is NOT the cause of this problem! Remember, test cases show the same video works perfectly under these circumstances:
In all of these cases, the video streams through the same equipment and LAN. Also, I used two different gigabit LAN chipsets, one on PCI and one on PCIe, one Intel and one Marvel, and the problem is identical.
The test case that disables pacer.sys is very interesting also.
OK people keep talking about gigabit being the problem, I've tried this without the gigabit switch and also forcing my nic to negotiate at 100 Mbps.
Has anyone who is having this problem tried using just a 100 Mb NIC? I didn't think of this till now, at least it owuld end the discussion one way or the other as to weather or not gigabit anything is related to the issue.
I'll try putting one of my old 3Com NICs back in my machine and see if that changes anything