The New Hotness

Literally.  It's like 66C up in there.

With dialog bubble.  Because G+ Creative Kit.

Sound card straddled by two GTX 570s.  HOT.  Literally.

But the crashing is gone with the replacement card.  And performance is much better with dual GTX 570s.  Now, I can move on.

Apparatus Update

Couple of quick notes:

• System has been behaving as intended (no black screen/sound loop crashes) since I replaced the GTX 570 video card with a GT 8800.  Hopefully, when CyberpowerPC sends back a new GTX 570, the problem will not return.  At this point, I am extremely leaning (99.999999999999%) towards a video card hardware problem causing my Skyrim issues.
• I'm using 12 Skyrim mods now (can't remember them off hand, but mostly graphics related) and it's working awesomely.  The only thing that would make it more awesome is if I could use the GTX 570 that I paid for. :-)  So, yes, anxiously awaiting for the RMA process.
• I'd love to use Skyrim UI since the inventory menus are so badly designed, but I don't want to deal with the hassle of continually updating SKSE.
• Eagerly waiting for the first Skyrim DLC
• Still looking for a good Bokeh mod that can be installed from the Steam Workshop.  I could use the Nexus ones, but I don't want the hassle, similar to maintaining SKSE.  Is that asking too much?

Bokeh

Liking CyberpowerPC

I'm really liking CyberpowerPC.  They consistently respond to my emails with quality customer service response, even if the answer is not positive (i.e. they don't do advance RMAs).  That, along with great support while purchasing and quick, quality shipping, will make me buy my next system from them.  After that I got this response to an email inquiring about the RMA process:

As we do not repair the video cards here at Cyberpower we will be replacing the video card all together. Once we receive your video card for replacement you will be notified by email.

Nice!  This should fit right in with validating if the problem is the video card or not.  In the meantime, I have a 8800GT 512MB video card and I'll be testing that to no end to see if it produces the same crash in Skyrim.

More On Video Card

After several more hours of testing and researching, I had a few more observations:

• Skyrim utilizes DirectX 9 and possibly some DirectX 10.
• Grid utilizes DirectX9 and possibly some DirectX 10.
• Battlefield 3 utilizes DirectX 11
• Batman: Arkham City utilizes DirectX 11

So far I've only experienced problems with Skyrim and Grid (the latter only in the past two days).  The crashes are the same: video feed goes out, sound loops, system unrecoverable.  This can happen after several hours or after just a minute (or even straight from power off, boot, game load, crash).  The usual suspects are out:

• Heat (measured over and over, max GPU is like 72C and max CPU is 68C.  This was extreme testing.  Normal operations is about 35C CPU and 42C GPU lower.
• Overclocking - the crashes are seen without overclocking, with overclocking and with underclocking.  It ain't the overclocking, okay?  I tested it.
• Malware - no malware on the system.
• System errors - NO ERRORS IN THE OS EVENT LOGS.
• Power Supply - replaced 750W power supply with a Silverstone 1000W power supply

Now, the fact that no issues are seen in Battlefield 3 and Batman and only the games NOT using DX11 led me to this article which describes how installing an older DirectX 9 game can muck up your DirectX 10 but leave DirectX 11 intact.  Weird, that sounds like my issue!

Quote from article:

Obviously, something is wrong with the DirectX 9 redistributable package. However, I don't think Microsoft can do anything about this since the DirectX 9 redistributable is everywhere and has been in use for ages. We just want to warn you about this problem so you won't need to scratch your head if your DirectX 10 game does not work. It could be because you just installed a DirectX 9 game. Try reinstalling the DirectX 10 redistributable.

Well, I tried the fix (install DirectX 10 runtime manually), but to no avail.  I still get the crashes in Skyrim and (now) Grid.  YARH (Yet Another Red Herring)?  Likely.

Frustrated and unable to get away from the observations:

* Crashes in DX9/10 games (Skyrim & Grid)

* Does not crash (yet) in DX11 games

* Just started crashing in Grid a few days ago

* No errors found using OCCT, Furmark, Heaven, etc, etc, etc

* No OS errors (this means the OS does not even catch the error, indicating it is likely NOT the video driver).

* Was able to change the crash slightly by disabling Timeout Detection Recovery (TDR).

All of the above points to a video card that is not behaving properly. 

Frustrated, I removed the Gigabyte GTX 570 from my system for packing up for the RMA and I found an older Nvidia Quadro FX 4600 sitting around, so I installed that.  I tried and tried to replicate the crashes, but I could not.  Changing the video card eliminated the observed crashes.  The logic is inescapable: it's not software, it's not my setup, it's not my configuration.  The only variable that changed was the video card itself.  Observation: video card is causing the problem. 

Guilty? We shall see.

It's weird that this would indicate that so many people have faulty video cards.  But then again, is it really so many?  In hindsight, no.  Sure, there are a lot of forum posts about it, but then again there are probably a lot of forum posts about....well, anything.  More than half of the posts tend to be trollers, leaving only a few with the same actual issues.

And then I remembered how my video card got a low rating (around 68%) for the ASIC in GPU-Z.  Perhaps it's true: Skyrim does push your hardware in ways other games and stress tests do not.  Here's another interesting fact: Had I not played Skyrim, I would have had zero issues with this system.  Well, at least I think so.

This is a very good video card.  When it works.

Hopefully this is the final test.  It is important to note that prior to the video card swap, I was already up to over 10 crashes in one day.  It was getting crazy.  After the swap and after several hours in Skyrim, loading saved games that would consistently crash, I got zero crashes with the older card.  

The GTX 570 now sits in a shipping box, ready for RMA.  However long it takes to get there and for CyberpowerPC to send me a replacement will hopefully be worth it if I've found the root cause of the problems for the last four months.  Now I just hope CyberpowerPC doesn't get the card, run Furmark against it and then send it back to me.  If that happens, I may just have to get another video card altogether.

"Video card, huh?  You were losing the video feed?  And then crashing?  Must be tough to be so frigging smart!"

TDR: Helpful. Next Stop: Video Card

After the TDR disabling, the crashes still happened, but more slowly.  In what previously was a quick video feed shutoff followed by a buzzing that was the last 1/2 second of audio looping now became a video feed shutoff, audio still playing with some distortion and the keyboard still responding.  I was unable to gain control though, since in about 10 seconds the keyboard became unresponsive and the audio eventually became the same buzzing.

This is at least something in the crashing that changed - and it means... something.  Previous changes were only on the frequency of the crashing, but this is the first time I was able to change the crashing method.  Another interesting change was I was able to get a single Windows Event Log entry:

Log Name: System
Source: nvlddmkm
Event ID: 14
Level: Error
Keywords: Classic
Task Category: None


Binary data:
In Words
0000: 00000000 00300002 00000000 C0AA000E
0008: 00000000 00000000 00000000 00000000
0010: 00000000 00000000


In Bytes
0000: 00 00 00 00 02 00 30 00   ......0.
0008: 00 00 00 00 0E 00 AA C0   ......ªÀ
0010: 00 00 00 00 00 00 00 00   ........
0018: 00 00 00 00 00 00 00 00   ........
0020: 00 00 00 00 00 00 00 00   ........


The description for Event ID 14 from source nvlddmkm cannot be found.
Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.


If the event originated on another computer, the display information
had to be saved with the event.


The following information was included with the event:


\Device\Video5
An uncorrectable double bit error (DBE) has been detected on GPU (03 03 03).


What this means to me is that the crash is centered on the video card.  Probably bad memory chips on the video card.  I ran GPU-Z and I got 68% on the ASIC - not clear on what that means, but the general consensus on the Internet is that the higher the percentage, the higher the quality of the video card build.

Shortly after this, with TDR still disabled, I saw the same crash in Grid.

Conclusion #1: Having TDR enabled is good, because it catches most crashes, preventing the system from going down.

Conclusion #2: Having TDR enabled is bad, because it hides these crashes from us and only when a crash is not recoverable (i.e. in Skyrim) do we think there is a problem.

Obviously #2 causes us to run down some crazy rabbit holes, while technically related, is really not something I can do anything about.

I've sent in an RMA request for the video card.  So we'll see what happens next...

Next Stop: TDR

Well, the crashes continue.  I've started a new Skyrim character and while it works fine 99% of the time, there's still the occasional jarring crash.  I've experimented with hardware, bought a new PSU, tried to work with Bethesda, all to no avail.

While I continued researching, trying to find commonalities between the reported crashes, I ran across something called "TDR" (Timeout Detection and Recovery) used in Microsoft Vista and beyond.

Right now, I'm going to experiment with disabling this feature.  The theory is that if the video card is busy, it cannot respond to the OS in the time and the OS will then send a restart command to the video card.  Right in the middle of Skyrim, this cannot be good.

To disable TDR in Windows Vista and Windows 7, either change or add the registry settings described in the link above.  For me, I set "TdrLevel" to "0" to disable it altogether.  I'd rather get a blue screen than what I'm getting now.

Bethesda Support Findings: null

Bethesda support was nice enough to exchange at least 11 emails with me, but alas, it led nowhere.  Here's their last response (emphasis mine):

The links you have provided are only anecdotal evidence at best. Whilst some of those people may be encountering the same symptoms as you are, ie a hardlock on their system, this does not mean that it has a common cause. For example some of the people in one of the threads you are providing have fixed their issue with hardware changes.

Furthermore, all of the tests you have suggested simply prove that your hardware works fine under the conditions that you are testing them under. Not all applications or tests will stress your system the same way. As an example, I had a similar issue to yours on a system I had been using with a different game. I was encountering random crashes and occasional lockups with one single game, all of the stress tests I did showed passed fine, and there were no temperature issues with the system. However as soon as the overclock on the system was removed, this game stopped crashing.

Whilst this is only anecdotal evidence, I hope that it provides some insight as to why I do not believe this to be a game/software related issue.

Now it may be that the game is simply encountering a crash, but because of the configuration of your system, the operating system is unable to handle the crash normally and so the system locks up and restarts. In this case it is very difficult for us to troubleshoot the problem, as we are unable to get any solid crash reports as your system is completely locking up.


With all that said, this game is still one of the best I've seen.  For completeness, here's my response to the above email:

I understand and appreciate your time on this.  So far the workaround for me when this does happen (and it's rarer now that I've been more frugal with binding spells and weapons/armor to the favorites) is to restart the system, load the last saved game and then make a few changes, create a new save file and then reload that new save file.
That seems to consistently get me past a crash point.  The overclock recommendation does not make a difference with my problem since it occurs with normal clock, overclock or underclock.  Believe me when I say that I've spent multiple hours troubleshooting this even before contacting support.

With that said, I'm continuing to enjoy the game (2nd character build now) and am looking forward to the DLC/expansion that's been in the news lately.  I've also began to experiment with the mods available on
Steam.  They definitely add to the replay value for this game, which despite the technical issues I've encountered, I still consider one of the best games made to date.

I'm assuming there was no progress on the save files I've sent previously?  Should I encounter another consistent, repeatable, crash, I'll try sending the save game file (along with my .ini files) in the hopes it will increase game stability.  Perhaps if a game debugging mode was made available/known, I could provide those logs to Bethesda
as well.

Thanks again for your time.

Status of the Apparatus

Skyrim being the appartus.

As of right now, Skyrim seems to be stable on my system.  Actually, not to put too fine a point on it, Skyrim is working with my current save file.  After all the testing I've done in the past several months, I've determined that my hardware is not the issue.  The problem that causes Skyrim to crash my system seems to be related to the game itself.  In interviews published last week, Bethesda's Skyrim director Todd Howard gave some insight to some of the issues that Skyrim developers have been aware of for some time now.  Specifically, it is NOT related to large save files:

"That’s the common misconception. It’s literally the things you’ve done in what order and what’s running. Some of the things are literally what spells do you have hot-keyed? Because, as you switch to them, they handle memory differently."

"We tried doing it through e-mail. We need to open the saved game comes up and look at it. We’ve got one guy who has seven dragons on the other side of the world, and a siege about to happen in this city and another 20 quests running. And, ok, this is what the game is trying to do and it’s having a hard time running that."

Not All Crashes Are Created Equal

After countless forum/website/blog reading, there seems to be several types of crashes.  Listed below are the ones I've categorized in my head as well as the theory I have and the suggested actions.  At the bottom of this post are some links to tools and some information on using them.  Note the information below (except the Skyrim specific ones) can be applied to any game or program that is giving you heat (sorry for the pun).

Crash to Desktop without Error Message - this happens while playing Skyrim and the TESV.EXE application just exists and dumps you to the desktop.
Theory: This is likely due to some kind of corruption with the game. 
What to do:

• Stop overclocking CPU, RAM and Video Card (just to be able to test with a default setup)
• Run RAM stress test to look for memory errors.  Replace defective RAM modules if necessary.
• Run Video Card stress test to look for artifacts, crashing, etc. 
• Upgrade Video Card drivers to latest STABLE NON-BETA release.
• Replace defective video card if necessary.
• Check for overheating

Crash to Desktop with Error Message - the error message is related to the video driver crashing and recovering
Theory: Something caused the video card driver to crash
What to do:

• Stop overclocking CPU, RAM and Video Card (just to be able to test with a default setup)
• Run Video Card stress test to test stability of video card and drivers
• Upgrade Video Card drivers to latest STABLE NON-BETA release.
• Check for overheating

Crash to Blue Screen - the standard Windows OS crash with a blue screen background with lots of data on the screen.  You all know this one, I'm sure.
Theory: Normally, a driver crashed, but the cause could be a bad driver or hardware the driver was talking to.
What to do:

• Stop overclocking CPU, RAM and Video Card (just to be able to test with a default setup)
• Identify the driver by looking at the Bluescreen or the dump files and test the driver and hardware associated with driver.
• Definitely get help on this one.
• Also, check to ensure that your Power Supply can keep up with your system.
• Check for overheating

Crash to Black Screen - this is what I've seen - it's a crash where the monitor loses signal and the sound loops and sometimes reboots your whole system.  Sometimes you have to power off manually.
Theory: Something in the game engine causes the video card hardware and/or RAM to access a part of memory that does not exist (computers don't like that). 
What to do:

• Stop overclocking CPU, RAM and Video Card (just to be able to test with a default setup)
• First, rule out the hardware:
• Run RAM Stress Test
• Run Video Stress Test
• Run CPU Stress Test
• Run the above overnight
• Check that your power supply can keep up with your system
• Check for overheating
• Fix any issues identified

If all the above are good, then do the following:

• Turn off the system and let it sit for a minute.  Also, to be sure, try removing the power cable for a few minutes to ensure that any artifacts are removed from memory.
• Power on the system.
• Start Skyrim.
• Turn off ALL Auto Saving.
• Reload your last saved game.
• Clean House!  This means remove any favorites that you don't use regularly, unbind unneeded hotkeys, get rid of ingredients and potions and anything else you are carrying around that you don't need (put them in a chest in your house or just drop them somewhere).
• Create a NEW save game.
• Exit Skyrim.
• Start Skyrim.
• Load that new save game.

Generally, if you suspect it's the last item above (i.e. a save file that is causing the crash), you'll want to keep that save file (make a copy) and then use that repeatedly.  For example, load it, make it crash so you know exactly what to do to make it crash and then restart, load it, make a change and see if it crashes at the same place.  If you get beyond it then do it again, don't do what you just did and see if it still crashes.  Then make the same change and see if it doesn't crash again.  In other words, make sure it's repeatable and demonstrable so you can pinpoint the change that makes it not crash.

Tools
 
Video card drivers - just go to your manufacturer's website and find the appropriate driver.  Be sure to do a "clean install" if possible - Nvidia driver installation has this option during the advanced install.

Hardware Testers:
Prime95 - Let this run at the highest settings overnight.  Be sure to test all of your RAM.  Any RAM issues should surface.
Linx - Another RAM tester.  Same as above - let it run against all your RAM overnight.
OCCT - http://www.ocbase.com/perestroika_en/index.php?Download - this tests multiple aspects of your system. Use it to test your GPU RAM and your CPU.  Again, let it run for a long time to encourage any hardware issues to surface.
Kombustor - this will test your GPU in DX9, DX10 and DX11 modes (test each one to be sure) - let this run overnight and any errors should surface.
Heaven - this one tests your game rendering.  Let this run overnight.  You can also choose DX9, DX10 and DX11 tests.  I suggest running them all, one after the other.

If you do the above tests, AT THE VERY LEAST you will feel good about your hardware being solid.  I've run the first four items AT THE SAME TIME for hours and hours and no issues surfaced for me and yet I experienced Black Screen crashes.

Monitoring for Overheating
I strongly recommend using HWiNFO (32 or 64 depending on your OS).  Start this and set it to log your stats to a file.   If you do crash, you can go back and check many, many things, like your CPU utilization, CPU temperature, System Temperature, GPU Temperature, GPU utilization, RAM usage, GPU RAM usage, etc, etc.  It's a very handy tool.  Feel free to send me your log files if you need help figuring out what it means.

Remember, this is an awesome game and there is a lot of mis-information out there.  When troubleshooting, you always want to get to the root cause of the issue.  Software developers will always blame your hardware first and hardware manufacturers will always blame the software.  Forget them - get to the truth yourself, learn something in the process and hopefully at the end you can enjoy your game.

Skyrim Update

Interesting thing happened when playing last night - at one point, just outside of Whiterun, I ran into an old Orc who wanted to "die a good death" and after I started to grant his which, the screen froze for 1/2 second, the sound started to stutter and it seemed like the start of a black screen/reboot crash, but then it recovered and no issue for the next four hours.

So what happened?  I think the game started to change all kinds of variables based on that decision to engage the Orc and that caused the stutter as things were loaded and changed.

I think if *THAT* situation happens AND there is a auto-save at the same time, things go downhill really fast and the whole system crashes.  Since I disabled auto-saving, then it didn't crash.  Sound reasonable?

I shouldn't say nothing else happened the rest of the session - after a period of time, I noticed a slight "heaviness" to the whole game.  Seemed a little jerky vs the normal super smooth.  It's barely noticeable.  But I saved the game, exited to desktop and restarted Skyrim and loaded the game and it was back to smoothness (this only took about 20 seconds for me).  I think this is just the engine having too much garbage (memory leak anyone?) in memory and reloading the program cleared it out.  Next time this happens, I will try just saving to a new save file and then loading it to see if the heaviness is lifted.

Anyway, that's my latest update.  And here's a puppy:

Could it really be this easy?

After going through the following, I still got two crashes with the same save game - seemed to have been alleviated with saving and loading a new game:

* Speedstep Settings (tried with off and on)
* Core Parking disabled
* Removed Sound Card
* Various other iterations of settings (compatibility mode, audio settings, admin mode, etc, etc)

Problem is: this "black screen/rebooting" thing ONLY HAPPENS IN SKYRIM.

Then I see a post from user "gjm777" on a couple of forums who advises to disable auto saving as he/she believes the problem is when loading a new area and autosaving at the same time.  This actually makes sense in light of the previous times I've seen this crash, including the latest crash: Creating a new save game seems to have made a difference.

Here's his post: http://forums.nexusmods.com/index.php?/topic/561338-black-screen-pc-crashlockup-fix-thread/

Here's what I just did:

1.Go to your %userprofile%\Documents\My Games\Skyrim directory.
2. Within the skyrimprefs.ini change the following lines as follows:

bSaveOnPause=0
bSaveOnTravel=0
bSaveOnWait=0bSaveOnRest=0
fAutosaveEveryXMins=0.0000

Current theory is that for some combination of hardware, some save games get corrupted, possibly due to timing issues (hey that sounds good), causing the engine to freak out.


New round of testing begins.  The soundcard removal and everything before that was a red herring.

Skyrim Black Screen/Reboot Crash Testing: So Far So Good

So far so good with finding the fix for Skyrim crashing/rebooting/black screen issue.  Approximately eight (8) hours of gameplay with no issues.  The current fix (if this lasts 40 hours, I will start calling it a real fix) is:

Disable Core Parking - this is a feature in Windows 7 and newer processors (some i5 and all i7 and higher Intel and recent AMD processors) that powers down unusued cores to save energy.  Tom's Hardware and Anandtech and other sites have looked at this and it is generally recommended to turn this feature off.  You can find instructions on doing so on the web and I have an earlier post that also has links to disabling it.  I am including it here because I know of at least one other user who was hard crashing (black screen/rebooting) and after they disabled Core Parking, no more crashes.

Removing Creative Labs X-Fi Sound Card - For me, disabling Core Parking made the game more stable, but crashes were still observed.  After I removed the CL X-Fi sound card and changed to the on-board audio, the crashes (so far) have gone away.

RECOMMENDATION: At this point, if you are having these crashes and have Windows 7, I would disable core parking and see if that works for you.  Additionally and regardless of OS version, if you have a dedicated sound card on the PCI-E slots AND have motherboard audio, try removing the sound card and switch to the motherboard audio (for me, this was simply powering down, opening the case, removing the sound card and then closing the case and then powering up).

I believe that the Core Parking plays havoc on the Skyrim processes - but at the very least it lowers performance.  As far as the sound card on the PCI-e bus - one observation I had was that my GTX 570 was showing as PCIe x8 instead of PCI x16.  This was also fixed after I removed the sound card, so my theory is that the sound card was somehow causing an issue with the PCIe bus.  Others have debugged and traced a Creative Labs sound card to show a bus-related issue, so it's entirely possible that the sound card + video card on the same bus is a bad thing, at least for some games (i.e. Skyrim).

Please let me know if this works for you.

1.4 Patch doesn't fix Resetting/Black Screen/Rebooting

Tested today and found that the resetting/black screen/rebooting still occurs with the 1.4 patch.  Next test item - completely removing the Creative SB X-Fi audio card.  Here's the sequence of events:

1. Encounter Reset/Reboot bug.
2. Restart PC
3. Do exact same thing (i.e. load the saved game), observe same result
4. Restart PC
5. Do exact same thing, observe same result
6. Change one variable (in this case, removing the audio card)
7. Do exact same thing.
1. If result is the same, return the change variable to previous value
2. If result has changed, note the change and continue testing until bug is encountered.
8. If the change lasts a week of playing (i.e. over 10 hours), then it's considered to be a success.

As of tonight, the current setup:

• Speedstep enabled
• Core Parking disabled
• Memory set to 1333MHz
• Turbo mode enabled
• Hyperthreading enabled
• Defaults in MB settings and GPU settings
• Creative audio card removed
• Using MB audio card

So far so good, it's at least running.  Testing continues on step 7-2...

Skyrim Save Games

UPDATE: It's quite possible that disabling Speedstep is overkill.  I'm having success re-enabling Speedstep and disabling CORE PARKING.

I was doing some reading and noticed that one of the issues with Skyrim that is routinely brought up is the "saved game issue," which is basically the idea that corrupt saved games or oversized saved games cause problems.  In almost any Skyrim troubleshooting forum thread, you will have someone post a suggestion to disable auto-save.

According to this post:

(Format is a bit different but as with Morrowind the .ess file is basically a "mod" loading up the changes you've done to the game and your character to try and describe it fairly simple.)

This looks like a reasonably accurate documentation for the Skyrim saved game file format.

What this tells me is that the author of the post is not a native English speaker and that the game basically starts up and then applies a series of variables to the engine to continue the game where you left off.  Something like:

1. Start at time 0, basic game start.
2. Apply the character variables (height, weight, color, attributes)
3. Apply the skills variables
4. Position player
5. Load the variables for the area completed (along with individual stats for each)
6. Load the character inventory
7. Fast forward time
8. etc
9. etc
10. Start the game

This is in line with what looks like (haven't tried yet) easy cheats, like giving yourself weapons and giving yourself extra perks through the tilde (~) console commands.  This also matches up with my experience where the crashes didn't happen right away, only after a period of time (i.e. variables changing).

I wonder if something in the game engine variables is bad with my saved games, which causes periodic crashes.  One way to test this is to clear out all the saved game info (have to Google how to do that) in order to "start from scratch" and then start a new character and see when the crashes start.  Or another way is to examine the contents of the saved game and somehow turn on debugging and see if there are any exceptions that are generated by the game engine (there has to be a debugging mode) when the saved game is loaded.

I'm assuming this is normal behavior for any game loading, but maybe the way this was implemented by Bethesda is buggy in some yet-unknown-configurations.  Maybe it's a combination of issues and I'm fumbling like a thumbless monkey trying to build a nuclear reactor.  Analysis continues....

Skyrim Crashes Revisited

UPDATE:  I'm having success re-enabling Speedstep and disabling CORE PARKING.

• 2011 DEC:
• Installed Skyrim, Steam patched it to the 1.3 version, supposedly fixing a bunch of things
• Skyrim works for a few days and then the dreaded "Reboot/Blinking Out" issue started.  In short, the entire system reboots.
• 2012 JAN:
• From late December to mid-January, troubleshooting in earnest.  All tests with other games, benchmarks, stress test, etc showed no issues. Roundup of testing here.
• Charted voltages, temperatures, etc and found no issues whatsoever outside of Skyrim rebooting the system.
• Bought and installed a 1000W Silverstone SST-ST1000-P PSU replacing the no-name 700W PSU that came with the system.  No change.
• Tried a series of changes, systematically testing each one.  The one that made a change was disabling Intel Speedstep.  Skyrim crashed no more.
• For about a week, no crashes and Skyrim is splendid!
• The other day, I got a desktop crash.  Restarted Skyrim and no further issues.
•  Last night, I got a reboot.  Not sure why, nothing had changed system-wise or application wise.  It's the same system that was running fine previously.  Reboots occured twice more and then stopped.  I doubt it's related, but I was at the mead poisoning mission just outside of Whiterun, in the dungeon just before combating the crazy conjurer that lived with the Skeevers.  I restarted the saved game and got beyong that part and no other issues for the next several hours.  (????????)

I am fairly confident (92%) that this is a problem with Skyrim programming.  I'm not bitter about having to buy a new PSU, because that just means ALL the critical components of my system are now high end pieces.  Also, if I ever decide to go dual or triple SLI, then I'm all set.

Skyrim being awesome

Skyrim not being awesome

Skyrim is an awesome game for sure, but these crashes are very, very frustrating.  It takes so much away from the game.  I really, really hope this gets resolved soon, either through Bethesda identifying an issue and releasing a patch or by me figuring out WTF is going on.