Showing posts with label Skyrim. Show all posts
Showing posts with label Skyrim. Show all posts

April 22, 2012

The New Hotness

Literally.  It's like 66C up in there.

With dialog bubble.  Because G+ Creative Kit.

Sound card straddled by two GTX 570s.  HOT.  Literally.
But the crashing is gone with the replacement card.  And performance is much better with dual GTX 570s.  Now, I can move on.

April 14, 2012

New GTX 570 = No More Skyrim Issues

Well at least the whole system reboot/black screen of death thing.

I got a new GTX 570 in the mail today, installed it and also tidied up the cables in the system.  So far so good.  Still a little nervous, half expecting the system to crash, but nothing like that so far!  Skyrim had a few issues, but nothing I couldn't recover from.  Which brings me to an interesting observation.  With the previous video card, I never saw Skyrim (the program) fail.  I would only see the whole system crash.  Now, I see the crash to desktop (rare) and the occasional video driver crashed thing (also rare).



One scary thing - about an hour into play, I heard a loud grinding.  Turns out the sound card was touching the fans on the GTX 570.  I couldn't do much more to separate them, so I ended up removing the sound card and switched to onboard audio.

See how close the sound card (top) is with the video card?  Yikes.

When I get the 2nd GTX 570 in, it will be really crowded in there with the sound card.  I'm thinking of gerry rigging some kind of wedge between the sound card and the video card, but in the meantime, I'll give the onboard audio a try.

It's been almost a week and so far so good.  Now if only the 2nd GTX 570 gets here soon...




April 04, 2012

Amazing Skyrim Story Engine

Don't want to reveal any spoilers, but when talking with a friend about Skyrim and he shared that at one point in the game, if you kill a main character in an abandoned shack, it will spawn a whole new set of quests.  When I was at that point in the game, it never even occurred to me to do that.  It's amazing that the Skyrim developers programmed that alternate storyline in.

March 27, 2012

Apparatus Update

Couple of quick notes:

  • System has been behaving as intended (no black screen/sound loop crashes) since I replaced the GTX 570 video card with a GT 8800.  Hopefully, when CyberpowerPC sends back a new GTX 570, the problem will not return.  At this point, I am extremely leaning (99.999999999999%) towards a video card hardware problem causing my Skyrim issues.
  • I'm using 12 Skyrim mods now (can't remember them off hand, but mostly graphics related) and it's working awesomely.  The only thing that would make it more awesome is if I could use the GTX 570 that I paid for. :-)  So, yes, anxiously awaiting for the RMA process.
  • I'd love to use Skyrim UI since the inventory menus are so badly designed, but I don't want to deal with the hassle of continually updating SKSE.
  • Eagerly waiting for the first Skyrim DLC
  • Still looking for a good Bokeh mod that can be installed from the Steam Workshop.  I could use the Nexus ones, but I don't want the hassle, similar to maintaining SKSE.  Is that asking too much?


Bokeh

March 20, 2012

Liking CyberpowerPC

I'm really liking CyberpowerPC.  They consistently respond to my emails with quality customer service response, even if the answer is not positive (i.e. they don't do advance RMAs).  That, along with great support while purchasing and quick, quality shipping, will make me buy my next system from them.  After that I got this response to an email inquiring about the RMA process:
As we do not repair the video cards here at Cyberpower we will be replacing the video card all together. Once we receive your video card for replacement you will be notified by email.
Nice!  This should fit right in with validating if the problem is the video card or not.  In the meantime, I have a 8800GT 512MB video card and I'll be testing that to no end to see if it produces the same crash in Skyrim.


March 18, 2012

More On Video Card

After several more hours of testing and researching, I had a few more observations:

  • Skyrim utilizes DirectX 9 and possibly some DirectX 10.
  • Grid utilizes DirectX9 and possibly some DirectX 10.
  • Battlefield 3 utilizes DirectX 11
  • Batman: Arkham City utilizes DirectX 11

So far I've only experienced problems with Skyrim and Grid (the latter only in the past two days).  The crashes are the same: video feed goes out, sound loops, system unrecoverable.  This can happen after several hours or after just a minute (or even straight from power off, boot, game load, crash).  The usual suspects are out:

  • Heat (measured over and over, max GPU is like 72C and max CPU is 68C.  This was extreme testing.  Normal operations is about 35C CPU and 42C GPU lower.
  • Overclocking - the crashes are seen without overclocking, with overclocking and with underclocking.  It ain't the overclocking, okay?  I tested it.
  • Malware - no malware on the system.
  • System errors - NO ERRORS IN THE OS EVENT LOGS.
  • Power Supply - replaced 750W power supply with a Silverstone 1000W power supply

Now, the fact that no issues are seen in Battlefield 3 and Batman and only the games NOT using DX11 led me to this article which describes how installing an older DirectX 9 game can muck up your DirectX 10 but leave DirectX 11 intact.  Weird, that sounds like my issue!

Quote from article:
Obviously, something is wrong with the DirectX 9 redistributable package. However, I don't think Microsoft can do anything about this since the DirectX 9 redistributable is everywhere and has been in use for ages. We just want to warn you about this problem so you won't need to scratch your head if your DirectX 10 game does not work. It could be because you just installed a DirectX 9 game. Try reinstalling the DirectX 10 redistributable.
Well, I tried the fix (install DirectX 10 runtime manually), but to no avail.  I still get the crashes in Skyrim and (now) Grid.  YARH (Yet Another Red Herring)?  Likely.

Frustrated and unable to get away from the observations:

* Crashes in DX9/10 games (Skyrim & Grid)
* Does not crash (yet) in DX11 games
* Just started crashing in Grid a few days ago
* No errors found using OCCT, Furmark, Heaven, etc, etc, etc
* No OS errors (this means the OS does not even catch the error, indicating it is likely NOT the video driver).
* Was able to change the crash slightly by disabling Timeout Detection Recovery (TDR).

All of the above points to a video card that is not behaving properly. 

Frustrated, I removed the Gigabyte GTX 570 from my system for packing up for the RMA and I found an older Nvidia Quadro FX 4600 sitting around, so I installed that.  I tried and tried to replicate the crashes, but I could not.  Changing the video card eliminated the observed crashes.  The logic is inescapable: it's not software, it's not my setup, it's not my configuration.  The only variable that changed was the video card itself.  Observation: video card is causing the problem. 

Guilty? We shall see.

It's weird that this would indicate that so many people have faulty video cards.  But then again, is it really so many?  In hindsight, no.  Sure, there are a lot of forum posts about it, but then again there are probably a lot of forum posts about....well, anything.  More than half of the posts tend to be trollers, leaving only a few with the same actual issues.

And then I remembered how my video card got a low rating (around 68%) for the ASIC in GPU-Z.  Perhaps it's true: Skyrim does push your hardware in ways other games and stress tests do not.  Here's another interesting fact: Had I not played Skyrim, I would have had zero issues with this system.  Well, at least I think so.

This is a very good video card.  When it works.
Hopefully this is the final test.  It is important to note that prior to the video card swap, I was already up to over 10 crashes in one day.  It was getting crazy.  After the swap and after several hours in Skyrim, loading saved games that would consistently crash, I got zero crashes with the older card.  

The GTX 570 now sits in a shipping box, ready for RMA.  However long it takes to get there and for CyberpowerPC to send me a replacement will hopefully be worth it if I've found the root cause of the problems for the last four months.  Now I just hope CyberpowerPC doesn't get the card, run Furmark against it and then send it back to me.  If that happens, I may just have to get another video card altogether.

"Video card, huh?  You were losing the video feed?  And then crashing?  Must be tough to be so frigging smart!"



March 17, 2012

TDR: Helpful. Next Stop: Video Card

After the TDR disabling, the crashes still happened, but more slowly.  In what previously was a quick video feed shutoff followed by a buzzing that was the last 1/2 second of audio looping now became a video feed shutoff, audio still playing with some distortion and the keyboard still responding.  I was unable to gain control though, since in about 10 seconds the keyboard became unresponsive and the audio eventually became the same buzzing.

This is at least something in the crashing that changed - and it means... something.  Previous changes were only on the frequency of the crashing, but this is the first time I was able to change the crashing method.  Another interesting change was I was able to get a single Windows Event Log entry:

Log Name: System
Source: nvlddmkm
Event ID: 14
Level: Error
Keywords: Classic
Task Category: None


Binary data:
In Words
0000: 00000000 00300002 00000000 C0AA000E
0008: 00000000 00000000 00000000 00000000
0010: 00000000 00000000


In Bytes
0000: 00 00 00 00 02 00 30 00   ......0.
0008: 00 00 00 00 0E 00 AA C0   ......ªÀ
0010: 00 00 00 00 00 00 00 00   ........
0018: 00 00 00 00 00 00 00 00   ........
0020: 00 00 00 00 00 00 00 00   ........


The description for Event ID 14 from source nvlddmkm cannot be found.
Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.


If the event originated on another computer, the display information
had to be saved with the event.


The following information was included with the event:


\Device\Video5
An uncorrectable double bit error (DBE) has been detected on GPU (03 03 03).


What this means to me is that the crash is centered on the video card.  Probably bad memory chips on the video card.  I ran GPU-Z and I got 68% on the ASIC - not clear on what that means, but the general consensus on the Internet is that the higher the percentage, the higher the quality of the video card build.

Shortly after this, with TDR still disabled, I saw the same crash in Grid.

Conclusion #1: Having TDR enabled is good, because it catches most crashes, preventing the system from going down.

Conclusion #2: Having TDR enabled is bad, because it hides these crashes from us and only when a crash is not recoverable (i.e. in Skyrim) do we think there is a problem.

Obviously #2 causes us to run down some crazy rabbit holes, while technically related, is really not something I can do anything about.

I've sent in an RMA request for the video card.  So we'll see what happens next...

March 16, 2012

Next Stop: TDR

Well, the crashes continue.  I've started a new Skyrim character and while it works fine 99% of the time, there's still the occasional jarring crash.  I've experimented with hardware, bought a new PSU, tried to work with Bethesda, all to no avail.

While I continued researching, trying to find commonalities between the reported crashes, I ran across something called "TDR" (Timeout Detection and Recovery) used in Microsoft Vista and beyond.

Right now, I'm going to experiment with disabling this feature.  The theory is that if the video card is busy, it cannot respond to the OS in the time and the OS will then send a restart command to the video card.  Right in the middle of Skyrim, this cannot be good.

To disable TDR in Windows Vista and Windows 7, either change or add the registry settings described in the link above.  For me, I set "TdrLevel" to "0" to disable it altogether.  I'd rather get a blue screen than what I'm getting now.

March 08, 2012

Bethesda Support Findings: null

Bethesda support was nice enough to exchange at least 11 emails with me, but alas, it led nowhere.  Here's their last response (emphasis mine):

The links you have provided are only anecdotal evidence at best. Whilst some of those people may be encountering the same symptoms as you are, ie a hardlock on their system, this does not mean that it has a common cause. For example some of the people in one of the threads you are providing have fixed their issue with hardware changes.

Furthermore, all of the tests you have suggested simply prove that your hardware works fine under the conditions that you are testing them under. Not all applications or tests will stress your system the same way. As an example, I had a similar issue to yours on a system I had been using with a different game. I was encountering random crashes and occasional lockups with one single game, all of the stress tests I did showed passed fine, and there were no temperature issues with the system. However as soon as the overclock on the system was removed, this game stopped crashing.

Whilst this is only anecdotal evidence, I hope that it provides some insight as to why I do not believe this to be a game/software related issue.

Now it may be that the game is simply encountering a crash, but because of the configuration of your system, the operating system is unable to handle the crash normally and so the system locks up and restarts. In this case it is very difficult for us to troubleshoot the problem, as we are unable to get any solid crash reports as your system is completely locking up.



With all that said, this game is still one of the best I've seen.  For completeness, here's my response to the above email:

I understand and appreciate your time on this.  So far the workaround for me when this does happen (and it's rarer now that I've been more frugal with binding spells and weapons/armor to the favorites) is to restart the system, load the last saved game and then make a few changes, create a new save file and then reload that new save file.
That seems to consistently get me past a crash point.  The overclock recommendation does not make a difference with my problem since it occurs with normal clock, overclock or underclock.  Believe me when I say that I've spent multiple hours troubleshooting this even before contacting support.

With that said, I'm continuing to enjoy the game (2nd character build now) and am looking forward to the DLC/expansion that's been in the news lately.  I've also began to experiment with the mods available on
Steam.  They definitely add to the replay value for this game, which despite the technical issues I've encountered, I still consider one of the best games made to date.

I'm assuming there was no progress on the save files I've sent previously?  Should I encounter another consistent, repeatable, crash, I'll try sending the save game file (along with my .ini files) in the hopes it will increase game stability.  Perhaps if a game debugging mode was made available/known, I could provide those logs to Bethesda
as well.

Thanks again for your time.

March 03, 2012

Bethesda Support

I've been exchanging emails with Bethesda support regarding the Skyrim issues I've been having.  They've been really patient with me and I really appreciate their support.  The latest exchange has involved them looking at my system configs.  Their current concern is my overclocking:

From Bethesda:

Given that your system is crashing at a very low level and it is restarting this does not really point to a great amount of system stability. It should be noted that the person in the youtube video who which you passed to us is also overclocking their system, which could point to a common demoninator.

My Response:

I'm not sure what to say other than I have the same amount of problems
when I'm not overclocking and even when I'm underclocking and even
when everything is set to auto.  I do agree with your statement in
general terms, but in my case, everything I've tried thus far has NOT
pointed to anything OTHER THAN a stable system.  I have not been able
to create a crash, either system or to desktop, or errors or anything
at all OTHER THAN in Skyrim. The ironic part is that I bought this
system specifically for Skyrim and had only Steam and Skyrim installed
when I first started having problems.  I did NOTHING to my system when
I started having problems, it was literally out of the box.
Troubleshooting started when I started crashing over and over and the
game became unplayable.

Here's a quick summary of what I've done to date:

* Reinstalled Skyrim from Steam
* Validated files in Steam
* Replaced power supply with a known good brand, high rated, over
powered (1000W)
* Reinstalled drivers (Video/System/etc) using a clean install
* Ran Prime95 (to test RAM) for 7 hours
* Ran Linx (to also test RAM) for 1 hour
* Ran OCCT (to test GPU RAM)
* Ran OCCT:CPU (to test CPU)
* Ran Kombustor
* Ran Heaven
* Ran Linx, OCCT and Kombustor AT THE SAME time
* Monitored for GPU, CPU and System Temperatures (logging to a file
every second) - no spikes seen just before crash in Skyrim.
* Removed Creative Labs audio card
* Underclocked CPU, RAM and GPU one at a time
* Disabled Intel speedstep
* Disabled CPU Parking

Non of these tests showed any problem whatsoever.  If the above tests
are not enough to test for system stability, then please let me know
what I can test for.

Again, there are NO ISSUES outside of Skyrim.  Skyrim could run for
four hours or 7 minutes before the system reboots.  At one point I had
a save file that would CONSISTENTLY cause this problem, but that file
could not be opened by your support staff and since the latest patch
would no longer crash for me.  This file would load inside of
Dragonsreach and I would turn around and exit the front door and the
system would reboot.  I did this multiple times in a row and then
loaded a different save game and didn't experience the reboot.  I've
also found that unbinding spells from favorites seemed to have caused
the reboots NOT to happen.  This clearly showed it was *something* in
the save file.  Again, I did this systematically: reboot/crash,
reboot/crash, reboot/crash, unbind spells, no reboot/crash, load
again, reboot/crash.  As I said earlier, this save file could not even
be opened by your support staff - it would cause the game to crash to
desktop.

I have zero issues in other games, namely Batman:Arkham City,
Battlefield 3 and Grid.  But I bought this system specifically to play
Skyrim and it's the only game causing me problems.

Others in the Internet have the same symptoms, as it's popped up in
multiple forums.  Clearly it's something that is specific since only a
minority of the millions of players out there have this problem.  I
know of two people personally who have had no issues in Skyrim.  As I
said above, overclocking is something I've done just recently because
all my testing showed that the system is solid - I can't find a
problem with it AT ALL, except for Skyrim crashing.  In other words,
crashing is isolated to Skyrim.

I read an interview with Bethesda's Todd Howard where he had stated
that things you do in the game cause issues, such as the PS3 lag
that's been widely reported.  I can't help but thing it's something
like that for my issue.  Now, I do understand about software
development and software issues, so what I'm looking for is this: what
is happening with my install/game that is causing this problem?  If
it's something like "binding spell x and spell y at the same time"
then it's not a problem to avoid that behavior.

I've maintained a blog with some graphs and more detailed information
of the troubleshooting I've done, including a red herring of disabling
Intel Speedstep (it did make it more stable, but I think it only
masked the problem) and disabling Windows 7 Core Parking.

This link shows you all the Skyrim related posts:

http://one-miguel.blogspot.com/search/label/Skyrim

But please, if you believe my system is unstable, let me know what
test to run so I can start to pinpoint the problem on my end, because
I've been trying to blame my system for the past three months but I
can't find a problem.

Thank you.

February 26, 2012

Skyrim Lag Over Time (PC) [Solved, Sort Of]

Skyrim has this, uh, feature that depending on how you play (what spells you use, what items you have, what order things are done, which butterfly flutters it's wings, etc, etc), sometimes your game will start to experience lag.  Specifically, input lag.  For me, it's barely noticeable, but I still do.

Anyway, if this happens to you, just do an "ALT-TAB" to go back to Windows and then ALT-TAB back to Skyrim.  You may need to do this a couple of times before the game actually comes back, but when it does, you are good to go.

"LAAAAAAAAAAAAAAAGGGGG!!!!!!"

February 21, 2012

Status of the Apparatus

Skyrim being the appartus.

As of right now, Skyrim seems to be stable on my system.  Actually, not to put too fine a point on it, Skyrim is working with my current save file.  After all the testing I've done in the past several months, I've determined that my hardware is not the issue.  The problem that causes Skyrim to crash my system seems to be related to the game itself.  In interviews published last week, Bethesda's Skyrim director Todd Howard gave some insight to some of the issues that Skyrim developers have been aware of for some time now.  Specifically, it is NOT related to large save files:

"That’s the common misconception. It’s literally the things you’ve done in what order and what’s running. Some of the things are literally what spells do you have hot-keyed? Because, as you switch to them, they handle memory differently."

"We tried doing it through e-mail. We need to open the saved game comes up and look at it. We’ve got one guy who has seven dragons on the other side of the world, and a siege about to happen in this city and another 20 quests running. And, ok, this is what the game is trying to do and it’s having a hard time running that."

Not All Crashes Are Created Equal

After countless forum/website/blog reading, there seems to be several types of crashes.  Listed below are the ones I've categorized in my head as well as the theory I have and the suggested actions.  At the bottom of this post are some links to tools and some information on using them.  Note the information below (except the Skyrim specific ones) can be applied to any game or program that is giving you heat (sorry for the pun).

Crash to Desktop without Error Message - this happens while playing Skyrim and the TESV.EXE application just exists and dumps you to the desktop.
Theory: This is likely due to some kind of corruption with the game. 
What to do:
  • Stop overclocking CPU, RAM and Video Card (just to be able to test with a default setup)
  • Run RAM stress test to look for memory errors.  Replace defective RAM modules if necessary.
  • Run Video Card stress test to look for artifacts, crashing, etc. 
  • Upgrade Video Card drivers to latest STABLE NON-BETA release.
  • Replace defective video card if necessary.
  • Check for overheating

Crash to Desktop with Error Message - the error message is related to the video driver crashing and recovering
Theory: Something caused the video card driver to crash
What to do:
  • Stop overclocking CPU, RAM and Video Card (just to be able to test with a default setup)
  • Run Video Card stress test to test stability of video card and drivers
  • Upgrade Video Card drivers to latest STABLE NON-BETA release.
  • Check for overheating

Crash to Blue Screen - the standard Windows OS crash with a blue screen background with lots of data on the screen.  You all know this one, I'm sure.
Theory: Normally, a driver crashed, but the cause could be a bad driver or hardware the driver was talking to.
What to do:
  • Stop overclocking CPU, RAM and Video Card (just to be able to test with a default setup)
  • Identify the driver by looking at the Bluescreen or the dump files and test the driver and hardware associated with driver.
  • Definitely get help on this one.
  • Also, check to ensure that your Power Supply can keep up with your system.
  • Check for overheating

Crash to Black Screen - this is what I've seen - it's a crash where the monitor loses signal and the sound loops and sometimes reboots your whole system.  Sometimes you have to power off manually.
Theory: Something in the game engine causes the video card hardware and/or RAM to access a part of memory that does not exist (computers don't like that). 
What to do:
  • Stop overclocking CPU, RAM and Video Card (just to be able to test with a default setup)
  • First, rule out the hardware:
  • Run RAM Stress Test
  • Run Video Stress Test
  • Run CPU Stress Test
  • Run the above overnight
  • Check that your power supply can keep up with your system
  • Check for overheating
  • Fix any issues identified
If all the above are good, then do the following:
  • Turn off the system and let it sit for a minute.  Also, to be sure, try removing the power cable for a few minutes to ensure that any artifacts are removed from memory.
  • Power on the system.
  • Start Skyrim.
  • Turn off ALL Auto Saving.
  • Reload your last saved game.
  • Clean House!  This means remove any favorites that you don't use regularly, unbind unneeded hotkeys, get rid of ingredients and potions and anything else you are carrying around that you don't need (put them in a chest in your house or just drop them somewhere).
  • Create a NEW save game.
  • Exit Skyrim.
  • Start Skyrim.
  • Load that new save game.
Generally, if you suspect it's the last item above (i.e. a save file that is causing the crash), you'll want to keep that save file (make a copy) and then use that repeatedly.  For example, load it, make it crash so you know exactly what to do to make it crash and then restart, load it, make a change and see if it crashes at the same place.  If you get beyond it then do it again, don't do what you just did and see if it still crashes.  Then make the same change and see if it doesn't crash again.  In other words, make sure it's repeatable and demonstrable so you can pinpoint the change that makes it not crash.

Tools
 
Video card drivers - just go to your manufacturer's website and find the appropriate driver.  Be sure to do a "clean install" if possible - Nvidia driver installation has this option during the advanced install.

Hardware Testers:
Prime95 - Let this run at the highest settings overnight.  Be sure to test all of your RAM.  Any RAM issues should surface.
Linx - Another RAM tester.  Same as above - let it run against all your RAM overnight.
OCCT - http://www.ocbase.com/perestroika_en/index.php?Download - this tests multiple aspects of your system. Use it to test your GPU RAM and your CPU.  Again, let it run for a long time to encourage any hardware issues to surface.
Kombustor - this will test your GPU in DX9, DX10 and DX11 modes (test each one to be sure) - let this run overnight and any errors should surface.
Heaven - this one tests your game rendering.  Let this run overnight.  You can also choose DX9, DX10 and DX11 tests.  I suggest running them all, one after the other.

If you do the above tests, AT THE VERY LEAST you will feel good about your hardware being solid.  I've run the first four items AT THE SAME TIME for hours and hours and no issues surfaced for me and yet I experienced Black Screen crashes.

Monitoring for Overheating
I strongly recommend using HWiNFO (32 or 64 depending on your OS).  Start this and set it to log your stats to a file.   If you do crash, you can go back and check many, many things, like your CPU utilization, CPU temperature, System Temperature, GPU Temperature, GPU utilization, RAM usage, GPU RAM usage, etc, etc.  It's a very handy tool.  Feel free to send me your log files if you need help figuring out what it means.

Remember, this is an awesome game and there is a lot of mis-information out there.  When troubleshooting, you always want to get to the root cause of the issue.  Software developers will always blame your hardware first and hardware manufacturers will always blame the software.  Forget them - get to the truth yourself, learn something in the process and hopefully at the end you can enjoy your game.


February 15, 2012

Skyrim Update

Interesting thing happened when playing last night - at one point, just outside of Whiterun, I ran into an old Orc who wanted to "die a good death" and after I started to grant his which, the screen froze for 1/2 second, the sound started to stutter and it seemed like the start of a black screen/reboot crash, but then it recovered and no issue for the next four hours.

So what happened?  I think the game started to change all kinds of variables based on that decision to engage the Orc and that caused the stutter as things were loaded and changed.

I think if *THAT* situation happens AND there is a auto-save at the same time, things go downhill really fast and the whole system crashes.  Since I disabled auto-saving, then it didn't crash.  Sound reasonable?

I shouldn't say nothing else happened the rest of the session - after a period of time, I noticed a slight "heaviness" to the whole game.  Seemed a little jerky vs the normal super smooth.  It's barely noticeable.  But I saved the game, exited to desktop and restarted Skyrim and loaded the game and it was back to smoothness (this only took about 20 seconds for me).  I think this is just the engine having too much garbage (memory leak anyone?) in memory and reloading the program cleared it out.  Next time this happens, I will try just saving to a new save file and then loading it to see if the heaviness is lifted.

Anyway, that's my latest update.  And here's a puppy:


February 12, 2012

Could it really be this easy?

After going through the following, I still got two crashes with the same save game - seemed to have been alleviated with saving and loading a new game:

* Speedstep Settings (tried with off and on)
* Core Parking disabled
* Removed Sound Card
* Various other iterations of settings (compatibility mode, audio settings, admin mode, etc, etc)

Problem is: this "black screen/rebooting" thing ONLY HAPPENS IN SKYRIM.

Then I see a post from user "gjm777" on a couple of forums who advises to disable auto saving as he/she believes the problem is when loading a new area and autosaving at the same time.  This actually makes sense in light of the previous times I've seen this crash, including the latest crash: Creating a new save game seems to have made a difference.

Here's his post: http://forums.nexusmods.com/index.php?/topic/561338-black-screen-pc-crashlockup-fix-thread/

Here's what I just did:
1.Go to your %userprofile%\Documents\My Games\Skyrim directory.
2. Within the skyrimprefs.ini change the following lines as follows:

bSaveOnPause=0
bSaveOnTravel=0

bSaveOnWait=0
bSaveOnRest=0
fAutosaveEveryXMins=0.0000
Current theory is that for some combination of hardware, some save games get corrupted, possibly due to timing issues (hey that sounds good), causing the engine to freak out.


New round of testing begins.  The soundcard removal and everything before that was a red herring.

February 10, 2012

Skyrim Black Screen/Reboot Crash Testing: So Far So Good

So far so good with finding the fix for Skyrim crashing/rebooting/black screen issue.  Approximately eight (8) hours of gameplay with no issues.  The current fix (if this lasts 40 hours, I will start calling it a real fix) is:

Disable Core Parking - this is a feature in Windows 7 and newer processors (some i5 and all i7 and higher Intel and recent AMD processors) that powers down unusued cores to save energy.  Tom's Hardware and Anandtech and other sites have looked at this and it is generally recommended to turn this feature off.  You can find instructions on doing so on the web and I have an earlier post that also has links to disabling it.  I am including it here because I know of at least one other user who was hard crashing (black screen/rebooting) and after they disabled Core Parking, no more crashes.

Removing Creative Labs X-Fi Sound Card - For me, disabling Core Parking made the game more stable, but crashes were still observed.  After I removed the CL X-Fi sound card and changed to the on-board audio, the crashes (so far) have gone away.

RECOMMENDATION: At this point, if you are having these crashes and have Windows 7, I would disable core parking and see if that works for you.  Additionally and regardless of OS version, if you have a dedicated sound card on the PCI-E slots AND have motherboard audio, try removing the sound card and switch to the motherboard audio (for me, this was simply powering down, opening the case, removing the sound card and then closing the case and then powering up).

I believe that the Core Parking plays havoc on the Skyrim processes - but at the very least it lowers performance.  As far as the sound card on the PCI-e bus - one observation I had was that my GTX 570 was showing as PCIe x8 instead of PCI x16.  This was also fixed after I removed the sound card, so my theory is that the sound card was somehow causing an issue with the PCIe bus.  Others have debugged and traced a Creative Labs sound card to show a bus-related issue, so it's entirely possible that the sound card + video card on the same bus is a bad thing, at least for some games (i.e. Skyrim).

Please let me know if this works for you.

February 05, 2012

1.4 Patch doesn't fix Resetting/Black Screen/Rebooting

Tested today and found that the resetting/black screen/rebooting still occurs with the 1.4 patch.  Next test item - completely removing the Creative SB X-Fi audio card.  Here's the sequence of events:

  1. Encounter Reset/Reboot bug.
  2. Restart PC
  3. Do exact same thing (i.e. load the saved game), observe same result
  4. Restart PC
  5. Do exact same thing, observe same result
  6. Change one variable (in this case, removing the audio card)
  7. Do exact same thing.
    1. If result is the same, return the change variable to previous value
    2. If result has changed, note the change and continue testing until bug is encountered.
  8. If the change lasts a week of playing (i.e. over 10 hours), then it's considered to be a success.

As of tonight, the current setup:
  • Speedstep enabled
  • Core Parking disabled
  • Memory set to 1333MHz
  • Turbo mode enabled
  • Hyperthreading enabled
  • Defaults in MB settings and GPU settings
  • Creative audio card removed
  • Using MB audio card
So far so good, it's at least running.  Testing continues on step 7-2...


January 30, 2012

Thoughts on Upcoming 1.4 Skyrim Patch

One theory I have for the Skyrim crashes that I and others are experiencing: Intel Speedstep and/or Core Parking powers down some cores prematurely, causing the Skyrim code to try and access a thread running on an affected core.  The core takes some time to power up, which the Skyrim code doesn't account for, which in turn causes the entire system to crash.  It's a theory based on observations, not necessarily in-depth knowledge of either the Skyrim code or how the technology works.  The observation is disabling Intel Speedstep and/or core parking (in other words, causing the cores to run at full speed all the time) increases stability (read: no crashing).  Of course there is the "not everyone is crashing" variable which may be due to the crashing systems being very fast (not everyone has the latest hardware), others having disabled core parking in the past (this affects older games as well) or having enough background processes to prevent the cores from powering down in the first place.  This is conjecture at this point, of course, but that's my theory in a nutshell.
The upcoming 1.4 patch should prove interesting.  One thing they supposedly fixed is optimizing the engine to use the hardware more efficiently.  This alone may fix the crashing problem.  There's definitely going to be a series of tests that I need to run after the patch comes out.

January 20, 2012

New Fangled Tech

Naturally, I'm researching Core Parking.  It's been around for a number of years, but during those same years, I've essentially "dropped out" of keeping up with technology.  My previous rig was a Dell with a dual-core (I forget which) and it worked fine for minor gaming.  It kept on working with newer games, so I never dug into the "new-fangled" stuff.  That system lasted me over five years.  Also, since the prices for components and custom machines have dropped significantly, actually building my own rig (or building for others) had become unreasonable: it's cheaper and easier to just order something that has what you need.  "Back in my day" you needed to build your own system if you wanted cutting edge technology without paying an arm and a leg (*ahem*Alienware*ahem*).  But with a new "custom built" rig that someone else (CyberpowerPC) built for me, I'm back waist-deep in researching the new-fangled technologies.

Not much has changed.  Well, a lot has changed tech wise, but the community seems to have stayed the same: just a few "serious" sites (Anandtech, Tom's Hardware, Overclockers.net), riddled with different groups of increasing population:

Very Few: The hard core geeks who can differentiate between different gate technology materials (I just made that up), drills holes into their CPUs and even solder on wires to get that last .5 Frame Per Second.  These guys will tear you a new one (by answering in detail or by dismissively giving a correct but unexpected answer) when you ask which processor is faster.

Minority: Somewhat serious folks who like to tinker, but are not drilling holes in CPUs (although when pushed, they will and then go into a depression).  They are up to speed with the latest marketing and specs and can generally work through tough issues, but also when to stop and accept a non-optimal solution.

Majority: Those who just want things to work and wander into various communities looking for help.  They eagerly want to solve their problem, but sometimes throw everyone off by asking how to edit the registry.

Of course, there are the usual nut-jobs as well and the angry ones who will rant and rave for pages and pages about how much they don't care about something.  These people tend to campaign for boycotts and demand something unreasonable, like a game company to call in programmers to fix their specific issue.

I like to think I fall in the minority because I know enough, but not enough

As I started with, I've been reading about Core Parking quite a bit and strongly suspect that this is the problem I (and others) might be having with Skyrim rebooting at random.  Consider this:

Note: my system has 2600k, Gigabyte MB, Gigabyte GTX 570, 16GB RAM

Known
  • Intermittent (sometimes after a minute, sometimes after an hour) hardware reset/crash/freeze.
  • Only with Skyrim
  • Works with other games
  • Works with Benchmarking tools (DX9, DX10, DX11) (over 7 hours running)
  • Works with stress testing tools (over 7 hours running)
  • Works with Benchmarking + Stress testing AT THE SAME TIME (Kombustor DX9 and DX10, 8X MSAA, 1920x1080, POST-FX, FULLSCREEN for about an hour)
  • Replaced Power Supply
  • Reinstalled Skyrim
  • Did the other things recommended by others on the 'net.
  • None of the above made a difference
  • No overheating
  • No error logs are generated - hardware just gets zapped
  • Unable to replicate rebooting/freezing outside of Skyrim
  • Disabling Speedstep made the system stable (one reboot in a week of playing for hours and hours a day).
  • Re-enabling Speedstep caused the crashes again.
  • Re-enabling Speedstep and disabling Core Parking seems stable (not fully tested yet).
  • Doing the above two seem to have worked for others (unsubstantiated reports only).
  • CPU/GPU load when playing Skyrim is very low on my system (settings are Ultra/maxed out).
  • Disabling Core Parking seems to be universally good for gaming (haven't found a negative post about it yet)
CPU Loads when running Skyrim

Unknown
  • Skyrim Creation Engine: is it built on Gamebryo? Why does it not use more cores?
  • Is Core Parking kicking in while playing Skyrim since the load is low?
  • Is the Skyrim engine somehow allergic to cores being powered down?
    • Skyrim works with Speedstep/C1E/C36/C6 State Support turned off (this prevents Core Parking)
    • Skyrim (seems to) work with Core Parking turned off and the above turned on
  • Out of the millions of Skyrim players, only a minority are having issues.  Is this because:
    • Only a few have high end processors + Windows 7 that support Core Parking?
    • Only a few have 16GB of RAM?
    • Only a few have low CPU/GPU use with Skyrim (most posts indicate that Skyrim is CPU intensive)?  This would mean my system is TOO FAST
    • Most have disabled Core Parking at some point in the past (I have a brand new box, nothing else loaded but games)?
    • Most have enough background processes to prevent Core Parking when playing Skyrim?
Anyway, some Core Parking things I found:
  • Works best in a server environment where there are cores that are truly not busy for long period of time
  • Does not save power if the system is under heavy load
  • Only in Windows 7, 8 and 2003, 2008
  • Impedes performance when enabled
  • Known to cause problems (Microsoft has a patch to disable it)

Testing to continue...

January 19, 2012

Skyrim Crash/Rebooting Fix: Disable Core Parking?

If you've read this blog before, you'll know that I've been having some issues with Skyrim on my new system.  Namely the problem was that Skyrim would cause the system to reboot intermittently.  It was very frustrating.  I had tried many, many things and the one that worked was disabling Speedstep.

However, this "solution" bugged me: it was not ideal and didn't explain the millions of other systems out there with Speedstep enabled that wasn't crashing.  In other words, it was not elegant or clean.  Of course, there are enough Internet posts that talk about the crashing, but certainly it is not as widespread as it should be if the problem was Speedstep.  Another problem was that my system was using power like it was still the 1990s!  This certainly goes against my green tendencies.  And then two days ago, I got yet another reboot in the middle of Skyrim.

So, I kept on searching when I stumbled across a series of posts and articles related to "Core Parking."  Having either never heard of this or assuming it's the same as Speedstep, I was surprised to learn that this was something that was found only the latest Intel processors (like mine) and is only found in the latest Windows OS (like mine).  I read through many many articles and it seemed this was indeed a likely culprit.

My theory goes like this: Skyrim's engine, the "Creation Engine" is not really new.  In fact, many others on the Internet postulate that it's simply built on the old engine, Gamebryo.  Gamebryo had some issues with Skyrim's predecessor, Oblivion.  Some research shows that the Creation Engine and Gamebryo share similar configuration files and in fact, the same behaviors.  Anyway, there's a lot to be said about Gamebryo vs. Creation that could fill pages and pages, but in short, I believe that "Creation" is a "new" engine that was built on the old one.  Gamebryo had problems with multi-core processors and it seems the Creation Engine also has at least some version of these problems.  Namely, not effectively using multi-cores and being single threaded.  In fact, some websites have shown that Skyrim does not run that much better with a quad-core vs a dual-core.  Something like 50% more performance from single-core to dual-core but only 3% from dual-core to quad-core.

Core Parking is essentially Windows deciding that if a core is not busy, it will power down that core to save energy.  If that core is needed for something, it wakes up in milliseconds.  My disabling Speedstep effectively prevented cores from being parked because they were running at full speed all the time anyway (although no load - it's quite possible that Core Parking relies on Speedstep to lower the core ratio/frequency and/or core voltage before deciding that the core is ready for 'parking').  At this point it's probably best to describe each:

Speedstep - this is a power-saving technology that reduces the frequency ratio of a core (mine goes from 34x to 16x @ 100MHz base, so 3.4GHz to 1.6GHz.  Additionally, it can reduce the voltage draw of a core as well.  The latest iteration also has a "Turbo" mode where the core can boost up to something like 42x (4.2GHz) if needed.  This is enabled/disabled in the BIOS.

Core Parking - also reduces power use, this basically disables a core.  This is a Windows thing.

Since I have a quad-core 2600K with multi-threading, Windows sees 8 processors.   During my test, when idle, with Speedstep on, Windows parked CPU1, CPU3, CPU5 and CPU6 and sometimes CPU7.  When I kicked up the load, the CPUs would get unparked and would kick in.

When Skyrim runs, it's not very processor intensive, at least not on my system.  The loads go up and down depending on what's happening.  Perhaps while in game, some cores are parked.  Perhaps combined with my 16GB of RAM and virtually nothing else running on my system, save for Steam, I'm not really needing a lot of cores while playing Skyrim.  My current theory is that neither the Gamebryo or the Creation Engine handles this very well with 4 cores and 8 "virtual processors."

I tested this by re-enabling all the defaults (i.e. Speedstep) and then disabling the Core Parking feature of Windows 7.  Note that there are many ways to do this.  I went with enabling the menu options and then setting minimum to 100% and maximum to 0% (see below).

Be sure to set "min cores" to 100% and "max cores" to 0%.
I recommend checking out the Microsoft way.  Anyway, it worked - I verified the Core Parking and Core Un-Parking effects using the built-in "Resource Manager" of Windows (scroll down for what it looks like).  Skyrim was played thoroughly for about and hour and DID NOT HAVE ANY CRASHES.  It's quite possible that this may be the "core issue" (pardon the pun).


It's still early in the testing, but at least I'm being somewhat green again, although not as green as when Core Parking is enabled.  But the power savings is minimal for me.  This is mostly intended for the huge server farms where some systems may stay idle for long stretches of time.

I'll give this at least a week of testing before I call it a win, but so far so good.  (I'll be out for a few weeks though, so no updates for a while.)   It's a nice discovery in any case and I learned way more about power savings than I intended to.  But now that I found it, I can't stop looking at it - kinda like the orb in Skyrim,

I found it, now I need to understand it before I can control it.

These are what parked cores look like in resource manager

After disabling core parking
Final note: After enabling the menu item, I made the changes and it immediately took effect, no reboot was necessary.  But, you probably should reboot anyway, just to make sure.  This is still Windows after all.  :-)

January 18, 2012

Skyrim Save Games

UPDATE: It's quite possible that disabling Speedstep is overkill.  I'm having success re-enabling Speedstep and disabling CORE PARKING.

I was doing some reading and noticed that one of the issues with Skyrim that is routinely brought up is the "saved game issue," which is basically the idea that corrupt saved games or oversized saved games cause problems.  In almost any Skyrim troubleshooting forum thread, you will have someone post a suggestion to disable auto-save.

According to this post:

(Format is a bit different but as with Morrowind the .ess file is basically a "mod" loading up the changes you've done to the game and your character to try and describe it fairly simple.)
This looks like a reasonably accurate documentation for the Skyrim saved game file format.

What this tells me is that the author of the post is not a native English speaker and that the game basically starts up and then applies a series of variables to the engine to continue the game where you left off.  Something like:

  1. Start at time 0, basic game start.
  2. Apply the character variables (height, weight, color, attributes)
  3. Apply the skills variables
  4. Position player
  5. Load the variables for the area completed (along with individual stats for each)
  6. Load the character inventory
  7. Fast forward time
  8. etc
  9. etc
  10. Start the game

This is in line with what looks like (haven't tried yet) easy cheats, like giving yourself weapons and giving yourself extra perks through the tilde (~) console commands.  This also matches up with my experience where the crashes didn't happen right away, only after a period of time (i.e. variables changing).

I wonder if something in the game engine variables is bad with my saved games, which causes periodic crashes.  One way to test this is to clear out all the saved game info (have to Google how to do that) in order to "start from scratch" and then start a new character and see when the crashes start.  Or another way is to examine the contents of the saved game and somehow turn on debugging and see if there are any exceptions that are generated by the game engine (there has to be a debugging mode) when the saved game is loaded.

I'm assuming this is normal behavior for any game loading, but maybe the way this was implemented by Bethesda is buggy in some yet-unknown-configurations.  Maybe it's a combination of issues and I'm fumbling like a thumbless monkey trying to build a nuclear reactor.  Analysis continues....