One theory I have for the Skyrim crashes that I and others are experiencing: Intel Speedstep and/or Core Parking powers down some cores prematurely, causing the Skyrim code to try and access a thread running on an affected core. The core takes some time to power up, which the Skyrim code doesn't account for, which in turn causes the entire system to crash. It's a theory based on observations, not necessarily in-depth knowledge of either the Skyrim code or how the technology works. The observation is disabling Intel Speedstep and/or core parking (in other words, causing the cores to run at full speed all the time) increases stability (read: no crashing). Of course there is the "not everyone is crashing" variable which may be due to the crashing systems being very fast (not everyone has the latest hardware), others having disabled core parking in the past (this affects older games as well) or having enough background processes to prevent the cores from powering down in the first place. This is conjecture at this point, of course, but that's my theory in a nutshell.
The upcoming 1.4 patch should prove interesting. One thing they supposedly fixed is optimizing the engine to use the hardware more efficiently. This alone may fix the crashing problem. There's definitely going to be a series of tests that I need to run after the patch comes out.
January 30, 2012
January 20, 2012
New Fangled Tech
Naturally, I'm researching Core Parking. It's been around for a number of years, but during those same years, I've essentially "dropped out" of keeping up with technology. My previous rig was a Dell with a dual-core (I forget which) and it worked fine for minor gaming. It kept on working with newer games, so I never dug into the "new-fangled" stuff. That system lasted me over five years. Also, since the prices for components and custom machines have dropped significantly, actually building my own rig (or building for others) had become unreasonable: it's cheaper and easier to just order something that has what you need. "Back in my day" you needed to build your own system if you wanted cutting edge technology without paying an arm and a leg (*ahem*Alienware*ahem*). But with a new "custom built" rig that someone else (CyberpowerPC) built for me, I'm back waist-deep in researching the new-fangled technologies.
Not much has changed. Well, a lot has changed tech wise, but the community seems to have stayed the same: just a few "serious" sites (Anandtech, Tom's Hardware, Overclockers.net), riddled with different groups of increasing population:
Very Few: The hard core geeks who can differentiate between different gate technology materials (I just made that up), drills holes into their CPUs and even solder on wires to get that last .5 Frame Per Second. These guys will tear you a new one (by answering in detail or by dismissively giving a correct but unexpected answer) when you ask which processor is faster.
Minority: Somewhat serious folks who like to tinker, but are not drilling holes in CPUs (although when pushed, they will and then go into a depression). They are up to speed with the latest marketing and specs and can generally work through tough issues, but also when to stop and accept a non-optimal solution.
Majority: Those who just want things to work and wander into various communities looking for help. They eagerly want to solve their problem, but sometimes throw everyone off by asking how to edit the registry.
Of course, there are the usual nut-jobs as well and the angry ones who will rant and rave for pages and pages about how much they don't care about something. These people tend to campaign for boycotts and demand something unreasonable, like a game company to call in programmers to fix their specific issue.
I like to think I fall in the minority because I know enough, but not enough.
As I started with, I've been reading about Core Parking quite a bit and strongly suspect that this is the problem I (and others) might be having with Skyrim rebooting at random. Consider this:
Note: my system has 2600k, Gigabyte MB, Gigabyte GTX 570, 16GB RAM
Known
Unknown
Testing to continue...
Not much has changed. Well, a lot has changed tech wise, but the community seems to have stayed the same: just a few "serious" sites (Anandtech, Tom's Hardware, Overclockers.net), riddled with different groups of increasing population:
Very Few: The hard core geeks who can differentiate between different gate technology materials (I just made that up), drills holes into their CPUs and even solder on wires to get that last .5 Frame Per Second. These guys will tear you a new one (by answering in detail or by dismissively giving a correct but unexpected answer) when you ask which processor is faster.
Minority: Somewhat serious folks who like to tinker, but are not drilling holes in CPUs (although when pushed, they will and then go into a depression). They are up to speed with the latest marketing and specs and can generally work through tough issues, but also when to stop and accept a non-optimal solution.
Majority: Those who just want things to work and wander into various communities looking for help. They eagerly want to solve their problem, but sometimes throw everyone off by asking how to edit the registry.
Of course, there are the usual nut-jobs as well and the angry ones who will rant and rave for pages and pages about how much they don't care about something. These people tend to campaign for boycotts and demand something unreasonable, like a game company to call in programmers to fix their specific issue.
I like to think I fall in the minority because I know enough, but not enough.
As I started with, I've been reading about Core Parking quite a bit and strongly suspect that this is the problem I (and others) might be having with Skyrim rebooting at random. Consider this:
Note: my system has 2600k, Gigabyte MB, Gigabyte GTX 570, 16GB RAM
Known
- Intermittent (sometimes after a minute, sometimes after an hour) hardware reset/crash/freeze.
- Only with Skyrim
- Works with other games
- Works with Benchmarking tools (DX9, DX10, DX11) (over 7 hours running)
- Works with stress testing tools (over 7 hours running)
- Works with Benchmarking + Stress testing AT THE SAME TIME (Kombustor DX9 and DX10, 8X MSAA, 1920x1080, POST-FX, FULLSCREEN for about an hour)
- Replaced Power Supply
- Reinstalled Skyrim
- Did the other things recommended by others on the 'net.
- None of the above made a difference
- No overheating
- No error logs are generated - hardware just gets zapped
- Unable to replicate rebooting/freezing outside of Skyrim
- Disabling Speedstep made the system stable (one reboot in a week of playing for hours and hours a day).
- Re-enabling Speedstep caused the crashes again.
- Re-enabling Speedstep and disabling Core Parking seems stable (not fully tested yet).
- Doing the above two seem to have worked for others (unsubstantiated reports only).
- CPU/GPU load when playing Skyrim is very low on my system (settings are Ultra/maxed out).
- Disabling Core Parking seems to be universally good for gaming (haven't found a negative post about it yet)
 |
| CPU Loads when running Skyrim |
Unknown
- Skyrim Creation Engine: is it built on Gamebryo? Why does it not use more cores?
- Is Core Parking kicking in while playing Skyrim since the load is low?
- Is the Skyrim engine somehow allergic to cores being powered down?
- Skyrim works with Speedstep/C1E/C36/C6 State Support turned off (this prevents Core Parking)
- Skyrim (seems to) work with Core Parking turned off and the above turned on
- Out of the millions of Skyrim players, only a minority are having issues. Is this because:
- Only a few have high end processors + Windows 7 that support Core Parking?
- Only a few have 16GB of RAM?
- Only a few have low CPU/GPU use with Skyrim (most posts indicate that Skyrim is CPU intensive)? This would mean my system is TOO FAST
- Most have disabled Core Parking at some point in the past (I have a brand new box, nothing else loaded but games)?
- Most have enough background processes to prevent Core Parking when playing Skyrim?
- Works best in a server environment where there are cores that are truly not busy for long period of time
- Does not save power if the system is under heavy load
- Only in Windows 7, 8 and 2003, 2008
- Impedes performance when enabled
- Known to cause problems (Microsoft has a patch to disable it)
Testing to continue...
January 19, 2012
Skyrim Crash/Rebooting Fix: Disable Core Parking?
If you've read this blog before, you'll know that I've been having some issues with Skyrim on my new system. Namely the problem was that Skyrim would cause the system to reboot intermittently. It was very frustrating. I had tried many, many things and the one that worked was disabling Speedstep.
However, this "solution" bugged me: it was not ideal and didn't explain the millions of other systems out there with Speedstep enabled that wasn't crashing. In other words, it was not elegant or clean. Of course, there are enough Internet posts that talk about the crashing, but certainly it is not as widespread as it should be if the problem was Speedstep. Another problem was that my system was using power like it was still the 1990s! This certainly goes against my green tendencies. And then two days ago, I got yet another reboot in the middle of Skyrim.
So, I kept on searching when I stumbled across a series of posts and articles related to "Core Parking." Having either never heard of this or assuming it's the same as Speedstep, I was surprised to learn that this was something that was found only the latest Intel processors (like mine) and is only found in the latest Windows OS (like mine). I read through many many articles and it seemed this was indeed a likely culprit.
My theory goes like this: Skyrim's engine, the "Creation Engine" is not really new. In fact, many others on the Internet postulate that it's simply built on the old engine, Gamebryo. Gamebryo had some issues with Skyrim's predecessor, Oblivion. Some research shows that the Creation Engine and Gamebryo share similar configuration files and in fact, the same behaviors. Anyway, there's a lot to be said about Gamebryo vs. Creation that could fill pages and pages, but in short, I believe that "Creation" is a "new" engine that was built on the old one. Gamebryo had problems with multi-core processors and it seems the Creation Engine also has at least some version of these problems. Namely, not effectively using multi-cores and being single threaded. In fact, some websites have shown that Skyrim does not run that much better with a quad-core vs a dual-core. Something like 50% more performance from single-core to dual-core but only 3% from dual-core to quad-core.
Core Parking is essentially Windows deciding that if a core is not busy, it will power down that core to save energy. If that core is needed for something, it wakes up in milliseconds. My disabling Speedstep effectively prevented cores from being parked because they were running at full speed all the time anyway (although no load - it's quite possible that Core Parking relies on Speedstep to lower the core ratio/frequency and/or core voltage before deciding that the core is ready for 'parking'). At this point it's probably best to describe each:
Speedstep - this is a power-saving technology that reduces the frequency ratio of a core (mine goes from 34x to 16x @ 100MHz base, so 3.4GHz to 1.6GHz. Additionally, it can reduce the voltage draw of a core as well. The latest iteration also has a "Turbo" mode where the core can boost up to something like 42x (4.2GHz) if needed. This is enabled/disabled in the BIOS.
Core Parking - also reduces power use, this basically disables a core. This is a Windows thing.
Since I have a quad-core 2600K with multi-threading, Windows sees 8 processors. During my test, when idle, with Speedstep on, Windows parked CPU1, CPU3, CPU5 and CPU6 and sometimes CPU7. When I kicked up the load, the CPUs would get unparked and would kick in.
When Skyrim runs, it's not very processor intensive, at least not on my system. The loads go up and down depending on what's happening. Perhaps while in game, some cores are parked. Perhaps combined with my 16GB of RAM and virtually nothing else running on my system, save for Steam, I'm not really needing a lot of cores while playing Skyrim. My current theory is that neither the Gamebryo or the Creation Engine handles this very well with 4 cores and 8 "virtual processors."
I tested this by re-enabling all the defaults (i.e. Speedstep) and then disabling the Core Parking feature of Windows 7. Note that there are many ways to do this. I went with enabling the menu options and then setting minimum to 100% and maximum to 0% (see below).
I recommend checking out the Microsoft way. Anyway, it worked - I verified the Core Parking and Core Un-Parking effects using the built-in "Resource Manager" of Windows (scroll down for what it looks like). Skyrim was played thoroughly for about and hour and DID NOT HAVE ANY CRASHES. It's quite possible that this may be the "core issue" (pardon the pun).
It's still early in the testing, but at least I'm being somewhat green again, although not as green as when Core Parking is enabled. But the power savings is minimal for me. This is mostly intended for the huge server farms where some systems may stay idle for long stretches of time.
I'll give this at least a week of testing before I call it a win, but so far so good. (I'll be out for a few weeks though, so no updates for a while.) It's a nice discovery in any case and I learned way more about power savings than I intended to. But now that I found it, I can't stop looking at it - kinda like the orb in Skyrim,
However, this "solution" bugged me: it was not ideal and didn't explain the millions of other systems out there with Speedstep enabled that wasn't crashing. In other words, it was not elegant or clean. Of course, there are enough Internet posts that talk about the crashing, but certainly it is not as widespread as it should be if the problem was Speedstep. Another problem was that my system was using power like it was still the 1990s! This certainly goes against my green tendencies. And then two days ago, I got yet another reboot in the middle of Skyrim.
So, I kept on searching when I stumbled across a series of posts and articles related to "Core Parking." Having either never heard of this or assuming it's the same as Speedstep, I was surprised to learn that this was something that was found only the latest Intel processors (like mine) and is only found in the latest Windows OS (like mine). I read through many many articles and it seemed this was indeed a likely culprit.
My theory goes like this: Skyrim's engine, the "Creation Engine" is not really new. In fact, many others on the Internet postulate that it's simply built on the old engine, Gamebryo. Gamebryo had some issues with Skyrim's predecessor, Oblivion. Some research shows that the Creation Engine and Gamebryo share similar configuration files and in fact, the same behaviors. Anyway, there's a lot to be said about Gamebryo vs. Creation that could fill pages and pages, but in short, I believe that "Creation" is a "new" engine that was built on the old one. Gamebryo had problems with multi-core processors and it seems the Creation Engine also has at least some version of these problems. Namely, not effectively using multi-cores and being single threaded. In fact, some websites have shown that Skyrim does not run that much better with a quad-core vs a dual-core. Something like 50% more performance from single-core to dual-core but only 3% from dual-core to quad-core.
Core Parking is essentially Windows deciding that if a core is not busy, it will power down that core to save energy. If that core is needed for something, it wakes up in milliseconds. My disabling Speedstep effectively prevented cores from being parked because they were running at full speed all the time anyway (although no load - it's quite possible that Core Parking relies on Speedstep to lower the core ratio/frequency and/or core voltage before deciding that the core is ready for 'parking'). At this point it's probably best to describe each:
Speedstep - this is a power-saving technology that reduces the frequency ratio of a core (mine goes from 34x to 16x @ 100MHz base, so 3.4GHz to 1.6GHz. Additionally, it can reduce the voltage draw of a core as well. The latest iteration also has a "Turbo" mode where the core can boost up to something like 42x (4.2GHz) if needed. This is enabled/disabled in the BIOS.
Core Parking - also reduces power use, this basically disables a core. This is a Windows thing.
Since I have a quad-core 2600K with multi-threading, Windows sees 8 processors. During my test, when idle, with Speedstep on, Windows parked CPU1, CPU3, CPU5 and CPU6 and sometimes CPU7. When I kicked up the load, the CPUs would get unparked and would kick in.
When Skyrim runs, it's not very processor intensive, at least not on my system. The loads go up and down depending on what's happening. Perhaps while in game, some cores are parked. Perhaps combined with my 16GB of RAM and virtually nothing else running on my system, save for Steam, I'm not really needing a lot of cores while playing Skyrim. My current theory is that neither the Gamebryo or the Creation Engine handles this very well with 4 cores and 8 "virtual processors."
I tested this by re-enabling all the defaults (i.e. Speedstep) and then disabling the Core Parking feature of Windows 7. Note that there are many ways to do this. I went with enabling the menu options and then setting minimum to 100% and maximum to 0% (see below).
 |
| Be sure to set "min cores" to 100% and "max cores" to 0%. |
It's still early in the testing, but at least I'm being somewhat green again, although not as green as when Core Parking is enabled. But the power savings is minimal for me. This is mostly intended for the huge server farms where some systems may stay idle for long stretches of time.
I'll give this at least a week of testing before I call it a win, but so far so good. (I'll be out for a few weeks though, so no updates for a while.) It's a nice discovery in any case and I learned way more about power savings than I intended to. But now that I found it, I can't stop looking at it - kinda like the orb in Skyrim,
 |
| I found it, now I need to understand it before I can control it. |
 |
| These are what parked cores look like in resource manager |
 |
| After disabling core parking |
Final note: After enabling the menu item, I made the changes and it immediately took effect, no reboot was necessary. But, you probably should reboot anyway, just to make sure. This is still Windows after all. :-)
January 18, 2012
Skyrim Save Games
UPDATE: It's quite possible that disabling Speedstep is overkill. I'm having success re-enabling Speedstep and disabling CORE PARKING.
I was doing some reading and noticed that one of the issues with Skyrim that is routinely brought up is the "saved game issue," which is basically the idea that corrupt saved games or oversized saved games cause problems. In almost any Skyrim troubleshooting forum thread, you will have someone post a suggestion to disable auto-save.
According to this post:
What this tells me is that the author of the post is not a native English speaker and that the game basically starts up and then applies a series of variables to the engine to continue the game where you left off. Something like:
This is in line with what looks like (haven't tried yet) easy cheats, like giving yourself weapons and giving yourself extra perks through the tilde (~) console commands. This also matches up with my experience where the crashes didn't happen right away, only after a period of time (i.e. variables changing).
I wonder if something in the game engine variables is bad with my saved games, which causes periodic crashes. One way to test this is to clear out all the saved game info (have to Google how to do that) in order to "start from scratch" and then start a new character and see when the crashes start. Or another way is to examine the contents of the saved game and somehow turn on debugging and see if there are any exceptions that are generated by the game engine (there has to be a debugging mode) when the saved game is loaded.
I'm assuming this is normal behavior for any game loading, but maybe the way this was implemented by Bethesda is buggy in some yet-unknown-configurations. Maybe it's a combination of issues and I'm fumbling like a thumbless monkey trying to build a nuclear reactor. Analysis continues....
I was doing some reading and noticed that one of the issues with Skyrim that is routinely brought up is the "saved game issue," which is basically the idea that corrupt saved games or oversized saved games cause problems. In almost any Skyrim troubleshooting forum thread, you will have someone post a suggestion to disable auto-save.
According to this post:
(Format is a bit different but as with Morrowind the .ess file is basically a "mod" loading up the changes you've done to the game and your character to try and describe it fairly simple.)This looks like a reasonably accurate documentation for the Skyrim saved game file format.
What this tells me is that the author of the post is not a native English speaker and that the game basically starts up and then applies a series of variables to the engine to continue the game where you left off. Something like:
- Start at time 0, basic game start.
- Apply the character variables (height, weight, color, attributes)
- Apply the skills variables
- Position player
- Load the variables for the area completed (along with individual stats for each)
- Load the character inventory
- Fast forward time
- etc
- etc
- Start the game
This is in line with what looks like (haven't tried yet) easy cheats, like giving yourself weapons and giving yourself extra perks through the tilde (~) console commands. This also matches up with my experience where the crashes didn't happen right away, only after a period of time (i.e. variables changing).
I wonder if something in the game engine variables is bad with my saved games, which causes periodic crashes. One way to test this is to clear out all the saved game info (have to Google how to do that) in order to "start from scratch" and then start a new character and see when the crashes start. Or another way is to examine the contents of the saved game and somehow turn on debugging and see if there are any exceptions that are generated by the game engine (there has to be a debugging mode) when the saved game is loaded.
I'm assuming this is normal behavior for any game loading, but maybe the way this was implemented by Bethesda is buggy in some yet-unknown-configurations. Maybe it's a combination of issues and I'm fumbling like a thumbless monkey trying to build a nuclear reactor. Analysis continues....
Skyrim Crashes Revisited
UPDATE: I'm having success re-enabling Speedstep and disabling CORE PARKING.
Skyrim is an awesome game for sure, but these crashes are very, very frustrating. It takes so much away from the game. I really, really hope this gets resolved soon, either through Bethesda identifying an issue and releasing a patch or by me figuring out WTF is going on.
- 2011 DEC:
- Installed Skyrim, Steam patched it to the 1.3 version, supposedly fixing a bunch of things
- Skyrim works for a few days and then the dreaded "Reboot/Blinking Out" issue started. In short, the entire system reboots.
- 2012 JAN:
- From late December to mid-January, troubleshooting in earnest. All tests with other games, benchmarks, stress test, etc showed no issues. Roundup of testing here.
- Charted voltages, temperatures, etc and found no issues whatsoever outside of Skyrim rebooting the system.
- Bought and installed a 1000W Silverstone SST-ST1000-P PSU replacing the no-name 700W PSU that came with the system. No change.
- Tried a series of changes, systematically testing each one. The one that made a change was disabling Intel Speedstep. Skyrim crashed no more.
- For about a week, no crashes and Skyrim is splendid!
- The other day, I got a desktop crash. Restarted Skyrim and no further issues.
- Last night, I got a reboot. Not sure why, nothing had changed system-wise or application wise. It's the same system that was running fine previously. Reboots occured twice more and then stopped. I doubt it's related, but I was at the mead poisoning mission just outside of Whiterun, in the dungeon just before combating the crazy conjurer that lived with the Skeevers. I restarted the saved game and got beyong that part and no other issues for the next several hours. (????????)
 | |
| Skyrim being awesome |
 |
| Skyrim not being awesome |
Skyrim is an awesome game for sure, but these crashes are very, very frustrating. It takes so much away from the game. I really, really hope this gets resolved soon, either through Bethesda identifying an issue and releasing a patch or by me figuring out WTF is going on.
January 17, 2012
Website Password Requirements
Increasingly, I have more and more passwords to remember. Mainly, this is due to different requirements (i.e. minimum length, upper/lower case, numbers, special characters, expiration every X months, etc). Unfortunately, because I am human, my passwords need to make some sense to me. I already have several "series" of passwords that I keep track of mentally. But, the problem is when faced with a login screen, I sometimes have a hard time remembering which password is for what (I'm still fighting the urge to write passwords down). If I could see what the password requirements are, I could make a reasonably successful guess at which password is what.
I believe I'm not the only one, so I'm going to start compiling a list of websites and their password requirements for reference. If you want me to add a site, please leave a comment with the attributes for that site and I'll add it.
I believe I'm not the only one, so I'm going to start compiling a list of websites and their password requirements for reference. If you want me to add a site, please leave a comment with the attributes for that site and I'll add it.
Site: amazon.com
Minimum Length: 6
Numbers Required: N
Upper/Lower Required: N
Special Chars Required: N
Expires: N
Site: google.com
Minimum Length: 8
Numbers Required: N
Upper/Lower Required: N
Special Chars Required: N
Expires: N
Site: newegg.com
Minimum Length: 6
Numbers Required: N
Upper/Lower Required: N
Special Chars Required: N
Expires: N
Other: No spaces, alphanumeric only
Site: isc2.org
Minimum Length: 8 - 20 characters
Numbers Required: Y
Upper/Lower Required: N
Special Chars Required: Y
Expires: N
Site: americanexpress.com
Minimum Length: 8 - 20 charactersNumbers Required: Y
Upper/Lower Required: N
Special Chars Required: Y (only %,&, _, ?, #, =, -)
Expires: N
Other: No spaces and NOT CASE SENSITIVE (!!)
NOTE: This means that they are storing the ACTUAL PASSWORD in their database and not using hashes. Ugh.
Site: Apple/iTunes
Minimum Length: 6 characters
Numbers Required: N
Upper/Lower Required: N
Special Chars Required: N
Expires: N
Site: dell.com
Minimum Length: 6 characters
Numbers Required: Y
Upper/Lower Required: N
Special Chars Required: N
Expires: N
NOTE 2: New requirements since they were hacked.
NOTE 3: Sends email notification of password change.
Site: zappos.com
Minimum Length: 8 characters
Numbers Required:Y/N (either this or Special Char)
Upper/Lower Required:Y
Special Chars Required:Y/N (either this or Numbers)
Expires: N
NOTE 1: Cannot use any of the previous 6 passwords. NOTE 2: New requirements since they were hacked.
NOTE 3: Sends email notification of password change.
Zappos Hacked
I woke up to this email on Sunday:
Here's the text of the email (bolding is mine):
After not panicking, I tried to figure out which password I used on the Zappos site, but alas I got this when I tried to access it:
Because clearly, blocking traffic from locations "outside the continental United States" is a security measure. *rolleyes* This is the Internet, folks - physical location is irrelevant. If I'm attacking a website in the US, the exploit packets will NOT be coming from outside the US, unless I want it to.
First, the bad news:
We are writing to let you know that there may have been illegal and unauthorized access to some of your customer account information on Zappos.com, including one or more of the following: your name, e-mail address, billing and shipping addresses, phone number, the last four digits of your credit card number (the standard information you find on receipts), and/or your cryptographically scrambled password (but not your actual password).
THE BETTER NEWS:
The database that stores your critical credit card and other payment data was NOT affected or accessed.
SECURITY PRECAUTIONS:
For your protection and to prevent unauthorized access, we have expired and reset your password so you can create a new password. Please follow the instructions below to create a new password.
We also recommend that you change your password on any other web site where you use the same or a similar password. As always, please remember that Zappos.com will never ask you for personal or account information in an e-mail. Please exercise caution if you receive any emails or phone calls that ask for personal information or direct you to a web site where you are asked to provide personal information.
PLEASE CREATE A NEW PASSWORD:
We have expired and reset your password so you can create a new password. Please create a new password by visiting Zappos.com and clicking on the "Create a New Password" link in the upper right corner of the web site and follow the steps from there.
We sincerely apologize for any inconvenience this may cause. If you have any additional questions about this process, please email us at passwordchange@zappos.com.
After not panicking, I tried to figure out which password I used on the Zappos site, but alas I got this when I tried to access it:
Because clearly, blocking traffic from locations "outside the continental United States" is a security measure. *rolleyes* This is the Internet, folks - physical location is irrelevant. If I'm attacking a website in the US, the exploit packets will NOT be coming from outside the US, unless I want it to.
Subscribe to:
Posts (Atom)