Old AVWW Top-Down Interior Generator Source Code Now Released

As promised, the source code (and runtime) for the old Interior Generator for the top-down version of AVWW has now been released.  If you want to read about what it is and what it does, please see the link above -- it's described in great detail.

And here is where you can download the source and runtime.

What's new in this version, since the blog post linked above?  Well, the cleanup processes have grown and grown, so there's something like 60 unique cleanup steps now.  It also builds even more varied interiors between the various interior styles, too.  I also started using chasms for the "remainders" where a wall wouldn't fit but I wanted to have something blocked off -- that worked out really well.

I'm sure there are still some little glitches here and there where something gets placed in a way that violates perspective, etc.  I had fixed most of those, but when we made the switch to a side-view this particular style of interior generation got tabled and so that was basically the end of this program for Arcen.

However, if you're making procedural interiors for a game that is top down, especially if it's tile-based with a faux perspective -- like so many pixelart games are -- then I welcome you to use this!  We'd appreciate credit if you feel like it, or at least a heads up that you're using it just for our own curiosity, but you're not legally obligated and it's not a huge deal to us either way.  Consider this released under the MIT license, and if you find it useful I'm glad.  Enjoy!

Fun with Windows (XP and 7).

Well, this morning right after checking my mail, I was logging on to my work VPN and found out that remote desktop would not open. Odd, I thought, but I was running north of 3GB of RAM and had not rebooted in over 6 months. So I figured I just needed to clear the air a bit, so to speak. Down went the various programs as I closed them, including Explorer.exe, which has an annoying memory leak in XP (at least when TortoiseSVN is present). Then tried to reopen Explorer.exe through Task Manager -- no go. Huh, that's odd, and not a good sign, I though. Time for a reboot for real, then.

The OS comes back up to a black screen, with a mouse cursor, and then just sits there blinking the hard drive light at me. Lovely. But I've had much more dire situations than this in the past, such as when a production server does something similar because of an extra bit of charge in the SCSI card. So I try all the usual tricks -- powering down and waiting 30+ seconds, running Last Known Good at least 1 more time than I had previously booted, then getting more concerned and trying to load Safe Mode (with the same result), then getting really concerned and deciding to do a Repair Install off of my slipstreamed XP SP2 disk... with the same result on reboot.

Oh, fun. What caused this? I have no idea. My hard disk and everything else seems to be okay. But it was clearly time for a reinstall. I've got an MSDN subscription, so I figured I'd install the Windows 7 64 RTM instead of just going back to XP 32. I have Vista 32 on a secondary tower that I use for testing with AI War, but I hadn't yet taken a look at Win7 directly. I'd heard good things from people I knew, though, and also felt like I couldn't stay on XP forever.

So, two hours later (the MS connection is really slow, even though my connection is really fast). I decided to just install Windows 7 onto the same single 500GB partition that I'd used for XP and all my data (gasp, I know). The install process for Windows 7 was slower than I expected, taking over an hour and half if I'm not mistaken (at least an hour) on my Q6600 quad. Hmm. Not off to too good a start.

Once the OS boots up, the first thing that I do is disable Aero (which is slow as well as nonfunctionally ugly in my opinion), returning to the Windows Classic theme. I also play around with the new taskbar a bit, returning it to something more like what I'm used to using (four rows tall, with autohide on). The quick launch bar is gone, but the new application pinning works really well. I decide to turn off the large icons and re-enable the text display (and turn off grouping). Perfect, it's now just about like what I prefer, and in some ways is a little bit better (I no longer have to install a little widget to allow me to drag around taskbar entries, for instance).

The next thing I notice is that explorer needs to reconfiguration to be a little more like what I like. Of course, the next next thing I notice is how insanely awesome the new Libraries feature is. How incredibly handy. By now I'm also marveling at how fast the (non-Aero) desktop experience is. It's about as fast as Windows XP, and tons faster than Vista, on this particular machine. The memory usage is higher, but since all 4GB of my RAM now actually registers with the OS (since I was on 32bit before, it capped out at around 3.3GB usable), it works out around the same anyway.

No problems with drivers, and all of the software that I have to install (including Visual Studio 2008) installs amazingly, blazingly fast. Huh. I've installed various versions of those pieces of software many times, on many machines, on several OSes over the years, and none of them ever installed even 1/8th as fast as it did on my machine here. Again, huh. That's pretty cool.

The rest of the software is similarly speedy and uneventful so far, and I'm down to only half a dozen more programs that I need to install, most of them noncritical for my daily work. I have noticed a few other changes, such as those to network places (meh), calculator (meh), remote desktop (wow!), windows find (wow!), IE8 (a hearty blah), picture viewer (pretty nice), and others. And the drivers seem pretty great for all my hardware, graphics card excepted.

In all this set me back around a day, though fortunately it was a day where I had a lot of meetings anyway so I could just do those on my other PC while the installs were running. So, not nearly as bad as some reinstalls I've had to do in the past. I find myself pleasantly pleased and surprised with Windows 7, too. It's not amazing in most respects so far as I can tell, but it's modern and unobtrusive, which is really what I was looking for. I always loved XP (and 2000 before it), after getting excited about (and burned by)about ME. But XP was starting to feel sort of dated, in areas that Win7 addresses. Simple usability improvements, like the taskbar, the dragging of windows to one side of the screen, the better truetype calibration and monitor sizing, the libraries, etc.

The sum of a hundred tiny improvements, which basic users may never even see, make me really happy with Win7.

Converting System.Drawing.Bitmap to XNA Texture2D

For those of you who visit this blog for art or writing topics, this is your fair warning: this post will hold no interest for you.

For those independent XNA game developers who find this post via Google, I hope I can help with a problem I've seen talked about in a few places on the 'net.

In the game I am currently working on, I have a need to load a bunch of bitmaps into memory, and then I need to turn those bitmaps into texture objects. I'm using a blend of GDI+ and XNA in my application, you see. Previously, when I was using MDX in conjunction with GDI+, this was no problem because there was a direct conversion available. Since XNA is dual-targeted at both the 360 and the Windows platforms, there isn't a conversion available.

On various forums I've seen solutions batted about relating to doing a per-pixel copy of the images from one format to the other (often using Bitmap.GetPixel, which is horribly slow -- you're much better off using Bitmap.LockBits, but even that is not nearly ideal).

The solution I have is simply relating to memory streams, since a Bitmap can be saved to a stream, and a Texture2D can be loaded from a stream. This approach might seem like a waste of memory, but it's the most processor-efficient way to do this. My game is able to process several dozen 28x28 images in under two seconds using this approach. The trick is to do the conversions just in little bits, as you need the images, rather than doing them all up front (which would take forever, and really give the garbage collector fits). I leave that part up to you. Here's the C# code for the actual conversion, which is quite simple:

Bitmap b = new Bitmap( nameOfFile );
Texture2D tx = null;
using ( MemoryStream s = new MemoryStream() )
{
b.Save( s, System.Drawing.Imaging.ImageFormat.Png );
s.Seek( 0, SeekOrigin.Begin ); //must do this, or error is thrown in next line
tx = Texture2D.FromFile( GraphicsDevice, s );
}

That's all there is to it!


(Added point of interest: It seems that XNA is unable to load GIF files -- presumably a licensing thing, knowing GIF -- but of course regular .NET is able to load those just fine. Using this sort of code provides a way for you to load GIFs or any other format that .NET supports but that XNA does not into XNA Texture2D objects. This is handy for me, because at present my project has... uh... just over 8,500 GIF files in it.)

Smooth Scaling Tiled Sprites In XNA

For those of you who visit this blog for art or writing topics, this is your fair warning: this post will hold no interest for you.

For those independent XNA game developers who find this post via Google, I hope I can help with a problem I've seen a lot of frustration on (and experienced frustration with myself). For background with the problem, see these posts (#1 and #2).

To summarize what you will find at those links, basically there is a "problem" when scaling images in XNA or DirectX wherein if you use the Sprite/SpriteBatch objects to draw a series of tiles, you'll get cruddy little lines, grids, or seams between many of your tiles. But the problem ONLY happens when zooming in (i.e., scaling 2D textures to a resolution higher than their native resolution), and it's fairly inconsistent. Sometimes half a pixel or so, sometimes up to a pixel, but never more, and it doesn't always even make a grid between every tile.

The main theories on this were that this was some sort of floating-point rounding error, or that this has to do with odd-sized textures (that perhaps are not powers of 2 -- mine, for instance, are 28x28), or that this was related to lacking the Clamp state of the u and v axes of the SampleState. Personally, my money was on some sort of "off by one" issue relating to zero-indexed widths and heights. None of these are correct.

As one enterprising programmer on the above links figured out, the real culprit is interpolation. By default, when scaling textures, Bilinear Interpolation is used to make it look nicer. If you are familiar with how that algorithm works, basically it's using a 2x2 grid of pixels adjacent to each target pixel, and blending them together. That works great in the middle, but at the edge of each tile there is nothing there -- each sprite tile is rendered independently (for the most part), which is why the black line creeps in. That line isn't a gap at all, it turns out, but rather a factor of the interpolation.

The quickest solution to this is to use point-based interpolation, which is basically no interpolation at all. In XNA, the C# code would be this:

sprite.Begin();
GraphicsDevice.SamplerStates[0].MagFilter = Microsoft.Xna.Framework.Graphics.TextureFilter.Point;

Problem solved, right? Well, yeah, but now we have a new problem -- without interpolation, your zoom is going to look awful. Programmers on the message boards had a bevy of potential solutions to this, some involving custom shaders, some involving replacing the SpriteBatch class, others involving manual edits to every image used in their game.

I have a vastly simpler solution (both in terms of programming effort/time, and in terms of processor time). Here's my rationale: this is an interpolation problem based on the fact that each tile is rendered separately, right? So the problem is not that we're scaling these tiles up, but rather that we're scaling them up one-by-one. If only there was a way to combine them all before rendering the current frame, and then scale them up together!

But wait, I hear you say -- something like that doesn't sound processor-friendly, right? That would basically double the amount of rendering we need to do, wouldn't it? If that's not what you were thinking, ten points for you for remembering that we're already doing that -- it's called the back buffer!

Since we're already rendering these sprites to the back buffer, then flipping them to the screen all at once, we've already got this pretty much handled. All we need to do is tweak the size of the back buffer before rendering, and it will automatically scale up to the view area -- perfect interpolation, great quality, no lines. The C# code looks like this:

float zoom = 0.8;
this.GraphicsDeviceService.ResetDevice( (int)Math.Round( this.ClientWidth * zoom ),
(int)Math.Round( this.ClientHeight * zoom ) );

I'm assuming here that you're using WinForms-hosted XNA code like from this example (http://creators.xna.com/en-us/sample/winforms_series1). If not, you'll have to fiddle with how to get this working in your environment. The basics are to set up a PresentationParameters variable with your desired width/height and to then do a graphicsDevice.Reset() and pass in that variable.

A few last points of interest:

- You'll notice that the zoom is inversed here. The zoom of 0.8 is actually equivalent to zooming in 1.2. The reason for the inversion is that we are shrinking our back buffer relative to the surface it will be rendered to.

- "this.ClientWidth" is assuming that you are calling this method from the Form, Panel, or whatever handle is your render target.

- As you may have already noticed, this method isn't compatible with your traditional "camera" approach, where you move a viewport relative to the world coordinates. To implement scrolling in your window (which is presumably the point here), you'll want to implement a global offset to your X and Y coordinates that are passed to SpriteBatch.Draw. NO NEED to do some massive global update of all your objects' coordinates as your window moves -- that's crazy. Leave your game world coordinates alone, and just do an offsetting of them as they are rendered in SpriteBatch.Draw. That way everything gets rendered efficiently, no massive updates are needed, and there isn't significant processor overhead incurred.

Happy coding!

Doctrine of Insufficient Adulation

Via Slashdot, I found an interesting article called Why Apple fans hate tech reporters. The basic premise is that when people are really rabid about an idea -- the merits of a certain product, religion, or political stance -- they perceive even-handedness as an insult. In other words, if I think you're completely wrong/evil/whatever, and someone else comes along and doesn't immediately leap to my side, I'll feel like they're siding with you.

Interesting. Disturbing. This is apparently just part of human nature, because I've certainly seen it before in life, though I'd never thought much about it until WSJ columnist Walt Mossberg pointed it out. It's particularly troubling for me, because I tend to avoid political conflict by remaining neutral. Go ahead, ask me what I think about abortion or Israel or Global Warming -- I'm not going to debate those things with anybody, at least not at present.

There are many conflicts in the world, a great many of which affect me, but I am not enough of an expert on them to feel like I can take a meaningful stance. On those issues with which I do have first-hand experience (the need for helmets on bicyclists, issues with violence in schools) I'll definitely speak out. I know it's more popular for people to have ardent opinions on anything and everything, but that's never sat well with me. I've always been a moderate, only taking one side or the other when I feel particularly informed or knowledgeable about the issue.

And now I discover that even moderation is likely to drag me into ideological scuffles I want no part of. How the heck does Switzerland do it? Oh, well, life wouldn't be interesting without conflict.

Manuscript Analyzer Online

The Windows version of Manuscript Analyzer has been getting some pretty good comments and use around the Internet, but one undercurrent that I've been encountering is that there are more writers who are Mac users than I had anticipated. To provide a solution for those users who can't use the Windows version, I've created Manuscript Analyzer Online.

This program is written entirely in JavaScript, and so works with Internet Explorer, Safari, Netscape, and FireFox without the need to install any software at all. The online edition contains all the features of the Windows edition, and is only very slightly slower to run.

The big concern that everyone is likely to have with this program, however, is its online nature. Is it stealing their manuscript, or transmitting it across the Internet where passing hackers might take advantage of it? The answer, of course, is no. Otherwise I wouldn't have put it up.

The program itself is about 24kb, and gets downloaded as a web page via your browser (like any other web page). Once it is downloaded, that's all that is transmitted: using JavaScript, the program analyzes the manuscript on your own computer, without any more interaction with my web server. This means that it also will work just as fast for dialup users as it does for users on cable or T1 (after that initial download), and that you don't have to keep an Internet connection open to use it.

Hopefully that alleviates any concerns that anyone has, but if there are other questions please do let me know and I'll be more than happy to address them. Also, I'm still quite interested in ideas for improvements/extensions to the program, so feel free to email me with those.

Speeding up FireFox

True to my computer/gamer nerd status (see last post), here's an interesting post: Crazy like a FireFox. It shows you to get even more speed out of the FireFox Internet browser. Not that I thought it was particularly slow before, but it does seem to render a smidge faster now...

Changing The Spacing Of Your Manuscript

Anne Mini recently did a great post on why it's a good idea to have two spaces after each period. This was actually not a habit I ever picked up, as I don't believe my typing teachers ever advocated this (my wife learned this in her schools, but I didn't -- maybe I just missed the boat). At any rate, when I heard of this preference a few months ago, I decided to make the shift to using the two spaces, and in order to do that I had to reformat my entire WIP manuscript.

Hours of tedious work? Not hardly! Using nothing more sophisticated than the "Find and Replace" function in MS Word, here's how to reformat your manuscript to add two spaces instead of one. Just follow these steps in order, and you should be good to go. It also works on manuscripts that are half and half -- part of it formatted with the correct two spaces, part of it formatted with only one.

Of course, as with any large-scale change you make to your manuscript, make sure and save a copy first! If something should go awry for whatever reason, you don't want it to happen to your only recent copy.

Now, on to the steps:

1.) Open "Find and Replace" in MS Word by hitting Ctrl+H in windows, or by looking under the Edit menu and choosing "Replace" (the menu location may vary slightly by Office version, especially in the 2007 edition, which moved the location of a lot of things -- I'm using Office 2003).

1.a.) Step two and below will use the character "_" to represent a literal space (since you otherwise couldn't see it), and square brackets [] to indicate the boundaries of the text you should be finding and replacing. When entering these values into your textboxes, omit the square brackets, and use a literal space where I show the underscore.

2.) Search for [._] and replace it with the value [.__]

3.) Search for [."_] and replace it with the value [."__]

4.) Search for [?_] and replace it with the value [?__]

5.) Search for [?"_] and replace it with the value [?"__]

6.) Search for [!_] and replace it with the value [!__]

7.) Search for [!"_] and replace it with the value [!"__]

8.) Search for [:_] and replace it with the value [:__]

9.) Search for [._._.] and replace it with the value [...]

10.) Search for [...__] and replace it with the value [..._]

11.) Search for [___] (three spaces) and replace it with the value [__] (two spaces)


Unless you have strange formatting or other interesting punctuation, that's it! And I'm sure that you can see how you can apply these general principles to other types of punctuation, or strange types of speech-delineation, like the carrots () that are used to identify thought-speech in Orson Scott Card's SPEAKER FOR THE DEAD.

The key is to run all your replaces that are like 2-8 first, so that you have at least the correct number of spaces after each sentence; then run numbers 9-11 to collapse any inappropriate extra spaces back down. If you're worried that things have gotten very out of whack, you can even run numbers 10 and 11 repeatedly, until they find nothing more to replace (that's actually not a bad idea to do, anyway, just to be sure).

I used this logic to successfully reformat my own manuscript several months ago, and I haven't found any problems with it in the interim. But I'm not the biggest expert on the rules of two spaces versus one, so if you see a hole in my logic, or if there's something I'm omitting, please let me know and I'll update the post. Happy formatting!


UPDATE 1: Someone pointed out on Anne's blog that this sort of logic incorrectly adds too many spaces after abbreviations. Since I didn't use any of those in my WIP, I didn't have that problem. But adding lines to search and replace things like [Mr.__] with [Mr._] would work, presuming you know, or can at least guess, all the abbreviations you used.

UPDATE 2: The list has been simplified a bit regarding quotes, thanks to a comment to this post.

UPDATE 3: Actually, that simplification was invalid, as another comment pointed out. In order to avoid adding extra spaces in cases like [what I think,"_said Tom.], the full steps above are necessary.

Restarting page numbering in word

This question gets asked a lot, so I thought I'd answer it here: "How do I omit the page number from my title page (plus however many other Table of Contents / Epigraph pages, or whatever else you've stuck in up at the start), and then have it start at page 1 at the start of my first chapter?"

1. Start out with no page numbering at all.

2. At the top of the page that you want to start numbering from, do the following: under the Insert menu in Microsoft Word, choose Break, and then choose one of the "section break" types.

2.a. If you've already got a page break properly in place, choose "Continuous" to not mess that up.

2.b. If you want to insert a page break with your new section, choose "Next Page" to do so.

3. Make sure your cursor is still on that same page that you want to start the numbering from, and then once again go to the Insert menu. Choose Page Numbers, use whatever settings you like, and then choose Format on the Page Numbers popup. Under the Formatting popup, change the "Continue from previous section" radio button to say "Start at 1" instead.

This divides your document into two "sections," each of which can have independent page numbering. You can actually use this same technique to divide your document as many times as you want, and have all sorts of different page numbering schemes (such as no page numbering at the very start, then roman numerals for an Introduction, then regular Arabic numerals for the main content, and then lettering for appendixes).

But, for manuscripts, remember that you'll just want to keep it simple and just have the Arabic numerals for the main content, starting in your first chapter.

My initial Mozy backup is finally complete

About a month ago I wrote a post on Mozy, a new backup tool that I had been using for a few weeks at the time. Well, I thought I'd post an update, since as of 5 hours ago, my initial backup set is finally complete.

Perhaps it seems egregious for an initial backup set to take a month and a half, but I was backing up 59.3 GB of data. About ten years worth of my computer art materials, (legal) music, home video clips, numberless digital photos taken by my wife, and of course my writing. The whole process took longer than it otherwise would have because I throttled it down to about 30 Kbps between 8:30 AM and 10:00 PM, so it was only able to run at a full (for my cable connection) ~390 Kbps during the night hours.

I must say, it's been a bit interesting checking in each day to see how much progress had been made. But still, I'm looking forward to having this just kick off at 2AM every night, and quickly back up whatever changes I've made during the day. That's what I got this for, after all.

If I had one main suggestion for Mozy regarding initial backup sets, it would be this: how about creating a service by which clients can email you DVDs for you to upload straight from your data center? The indexes would certainly be a problem since you normally map everything to the hard disks, but if you made it so that Mozy had the option to do an initial backup set to a set of DVDs, then you could embed the indexes right on the DVDs and easily read them when the clients send them in.

This, of course, would take development time away from your main efforts, and would also require ongoing man hours for staff to handle the DVDs as they arrive. So I wouldn't expect this service to be free -- perhaps $5 per DVD would work for all parties. Whatever covers your costs while making it easier/faster for clients to do that initial backup set. Although, I can see where that added service might cause capacity planning problems in your data center, because your volume of data could quickly jump upwards when a lot of DVDs come in, whereas right now every individual client stream arrives at a fairly predictable trickle.

There are always so many different concerns when you have a centrally hosted software service like this. Part of what professionally interests me in that model, I guess.

Talk about a wate of paper

Check out this article on the bills that AT&T sends to iPhone customers. Apparently some of these monthly bills were as large as "small novels." That sounds like gross exaggeration, but, looking at the pictures, it doesn't seem like that is the case at all. Makes you feel a little bit better about the volume of paper used for manuscripts in the publishing industry, doesn't it? At least all that paper actually has a point...

Mozy

A few weeks ago, a client told me about a program called Mozy. It was so inexpensive ($5/month) that I signed up right away, and I've discovered that this program is really a terrific value. It was recently featured in PC Magazine, I'm told, but I haven't subscribed to that particular publication for a number of years now (too consumer-oriented; I've gotten so I much prefer TechRepublic).

I did a popular post on Data Loss and Backup Solutions a few months ago, and I think that this particular product is an excellent solution to the backup issues I outlined before. This is very similar to the more-expensive tools from Connected TLM that I have used in the past, but this tool is much simpler and more cost-effective. They even have a version that is completely free for up to 2GB of data, which should be more than enough if all you're looking to back up is your writing materials. I have a a lot more data than that to back up (58GB for now), so I'm doing the paid account. If you don't already have a backup solution (or aren't happy with the one you do have), check this one out and see what you think.

What's the longevity of blogs?

Today I found this interesting post over at TechRepublic. From the post:

Security expert and tech curmudgeon Bruce Sterling famously quipped at this year’s South-by-Southwest conference that “I don’t think there will be that many [blogs] around in 10 years. I think they are a passing thing.”

Interesting. But I agree more with "The Trivia Geek," the author of the post, who puts forth his own theory on why blogs will, in fact, survive. There probably won't be nearly so many blogs, since those participating purely because of blogs' current fad status will likely move on, but good blogs will likely always remain until another tool comes along that is better able to "let writers write."

Tech Q&A: How do I enter less/greater than in blog comments?

I've seen this question various places around the blogosphere, so I thought I would address it as the first in a new "Tech Q&A" series. If you've ever tried to post the "<" or ">" characters in blog comments, you've undoubtedly found that these symbols (and anything between them) mysteriously disappeared. Why is that?

Well, when you type in your comments you are entering what is referred to as "plain text." That means that, while it's still in the comments box, your browser views whatever you type in as being literal Roman characters or symbols. Something for humans to read, in other words. When you type left or right carats here, there's no problem.

The problem arises when that plain text is submitted to blogger. Blogger no longer treats it as plain text, but rather starts treating it as HTML markup. This is actually very helpful, because it let's you embed links and images in your comments, and/or allows you to italicize or bold words using those tags.

The only problem with this approach is that if the commenter has entered certain characters -- most notably the left or right carat -- then content after the < will disappear. If there is a > that follows it, then content will become visible after that. I called this a problem, but really this is just how the Internet works in such situations; I don't think that there is much blogger could do differently save possibly not allow HTML in comments at all (which would be bad), or load their rich text editor on the comments page (which would be bulky, and thus undesirable). So I think that they're doing about the best they can reasonably do at this point.

What if you really need to display that < or >, though? For instance, what if you're writing out a math equation or trying to show someone else how to create some HTML code? Fear not, the browser programmers of yore thought of this long ago. There is a special character sequence that tells the browser to show a left or right carat to the user, but that carat will be ignored by the browser. My recent link to the ASCII Codes reference includes not only these sequences, but many more.

Here are the relevant codes:

When you want to show a literal "<" in the posted comments, type &lt; instead.
When you want to show a literal ">" in the posted comments, type &gt; instead.

Those character codes probably seem incomprehensible, so let's break them down. First, there is an opening character that tells the browser that a character code is coming; that's the ampersand. Then comes the actual character code itself, which is specific to the character you want to print, and then comes the closing character that tells the browser that your character code is done; that's the semicolon.

In the specific case of the above examples, the "lt" stands for "less than," and the "gt" stands for "greater than." Easy to remember, right? Feel free to ask for further clarification in my blog comments here, or even to try our the character codes there for yourself if you want to. Also, if anyone has topics that they'd like to see in a future Tech Q&A feature, please send them in.

The Future?

As a science fiction writer, not to mention as a computer programmer, I find the future to be a topic of great interest. Most futurists are concerned with grand depictions of planet colonization, advanced AI, robots, et cetera -- and while I do find these topics of interest, they seem a little distant to me. I don't realistically expect to see extreme advances in any of the preceding areas within my lifetime. I mean, don't get me wrong -- I'd love to see major advances in those fields, but I just have a hard time really believing I will.

This cynicism regarding technological advancement is what makes Hugo award winner Charlie Stross's predictions for the near future so compelling to me. In his predictions, Charlie intentionally doesn't account for the emergence of any new technologies, or for major breakthroughs in any of the leading-edge fields of today (genetics, nanotechnology, et cetera). Instead he focuses on following existing trends with existing technologies, such as computers and the Internet.

Part of his conclusion:

Meet your descendants. They don't know what it's like to be involuntarily lost, don't understand what we mean by the word "privacy", and will have access (sooner or later) to a historical representation of our species that defies understanding. They live in a world where history has a sharply-drawn start line, and everything they individually do or say will sooner or later be visible to everyone who comes after them, forever. They are incredibly alien to us.

Is anyone else reminded of the Hive Queen from ENDER's GAME? As alien as the lives of modern humans would be to world citizens from even a few hundred years ago, that's how alien the lives of our descendants are likely to be -- with or without hover cars and relativistic space flight. I've never thought of the evolution of our society in quite these terms before, but if Charlie Stross is right, the future might just be closer than we think (and different than we expect, as always).

Ne quid nimis

Blogging is addictive. And, as is the case with most addictions, that's not a good thing. Life is already too full of things to do . If you've been wondering where I was for the past week, perhaps you thought that I was:

A) Dealing with personal life issues.
B) Very busy with day-job work.
C) Working on my actual novel.
D) Trying to relax at least a little.
E) All of the above.

As is far too often the case with multiple choice questions, the answer is "all of the above." Though sadly not as much of answer C as I would have preferred. Lot's of planning for my manuscript, a fair bit of editing, and a little bit of writing. I'll have more on my progress sometime soon, but let's leave it at that for now.

To be clear: I have no plans for giving up my blog, but I'm also not going to let it dominate my evenings like I did for a while there. I'll continue to try and have a new post at least once a week, and many weeks may see (vastly) more posts than that, but I'm not going to adhere to any particular schedule. If you read a lot of blogs, my best advice is to subscribe to RSS feeds (that link is to mine). It's the easiest way to get notified whenever a blog author makes an update to his/her blog. That way you don't keep checking back every day, only to be disappointed whenever the daily offering isn't there. It's the concept of push technology at its best!

I know I've said things like this before, but this time I'm going to try stick to it. My addictions are collecting things and (apparently) blogging-related activities such as posting far too many comments elsewhere in the blogosphere, but I will overcome them! At least I don't have any of the really nasty habits--no smoking, no drugs, and alcohol only in extreme moderation. I suspect we all have some nontraditional addictions in our lives, and if those are preventing us from writing as much as we'd like, well, we ought to at least evaluate that.

"In all things moderation." --Publius Terentius Afer

On Data Loss and Backup Solutions

Ever had your entire book and all your notes disappear before you finished writing it? I really hope the answer is no--and let's keep it that way. My day job is as a programmer, but that also includes a lot of systems admin work. I'm used to being required to have lots of redundancy and backups in my computer systems, but this isn't an issue that your average home user is going to think much about. Total losses are rare enough that you don't hear about them too often, but they do happen, and perhaps more than you might think. Here's some information on how you can best protect yourself.

Most users know deep down that they should be doing backups, and the most common solution is CDRs or DVDRs. Problem is, this kind of backup is a bit of a pain to do, which makes users less likely to actually do it. Odds are that you've never lost any significant data, and so your perceived risk of losing any in the future is rather low. But computer problems of all bents happen all the time, and even if you religiously back up every week or every month, that may not be nearly enough.

How would you feel about losing a week or month of your work? I lost three days' worth of writing once, and that was terrible. Ideally you want a backup solution that works in something approaching real-time, so that whenever you do some work it gets backed up. This is what your corporate IT departments have been doing for years, and it's not as out of reach as home users seem to think.


Risks
Let's first consider the main risks, and then talk about solutions for each one.

Theft is probably the absolute worst thing that can happen to your computer, because there is much less chance of recovery. And while theft of PCs isn't very common, a surprising number of laptops get stolen. Corporate IT departments have all kinds of policies in place on what data has to be encrypted on employee laptops, and what data can't be put on those laptops at all, because they are aware of this risk.

Obviously you can try to take steps to make sure that your computer isn't stolen, such as putting cable locks on your laptop when you're in a public place (trust me, this is not as ridiculous as it might sound at first), but you also have to consider the fact that your computer might get stolen no matter what you do. And the only way to solve this problem is to have a copy of your data that is not on your local computer.

Fire is another extremely bad way to lose your data. Even if you've been backing up to CD, or copying data between your laptop to your PC, if your house burns down with everything in it, that's it for your book.

Even if you have a fireproof safe, most of those are only rated for 30-40 minutes or so, and don’t always do the best with plastic media inside. The ones that I linked to above are really expensive and do a better job, but they still won't last forever in the middle of a raging inferno. Having a good fireproof safe might well work, but it isn't guaranteed. Lowes has a wide range of prices for fireproof safes, as do many other retailers, and the smallest of these are affordable but relatively ineffective.

The only foolproof option in the case of fire is to have another computer somewhere off site that you can back up to.

Mechanical Failure of hard disks is a much more common cause of data loss than anything mentioned thus far. If your hard drive physically breaks for whatever reason, there are places that have forensics hardware/software to get that data off your dead drive. Prices range widely, as does quality of service, but you might be able to get your book back for "only" a few hundred dollars. The best solution here is to always keep your data on more than one hard disk.

Power Surges are even more common than mechanical failure. Hopefully everyone has their computers on surge protectors, but the quality of that surge protector is an important consideration. Some of the inexpensive ones just don't do a very good job. If a power surge does hit your computer, these days the most likely casualty is going to be your computer's power supply, which costs $30-$50 in a PC to replace and isn't going to lose any of your data. But that isn't to say that your hard drive is immune by any stretch. Make sure that your computer has an adequate surge protector.

Even better, you could get a battery backup unit (also known as a Uninterruptible Power Source or UPS). The smallest of these run for about $40, and will provide you with great surge protection as well as about 10-15 minutes in which to safely shut your computer down if your power does go out.

Improper Shutdown and Jostled Disks used to be a big problem for hard drives, but they aren't so much of an issue with modern drives. You still shouldn't hit or move your computer while it is on, and you should try to avoid turning off the power to your computer while the hard drive is actively writing, but the catastrophic failure that was common in such cases in the 80's and 90's isn't as much of an issue these days. Back then the armature inside the hard drive was more free moving, and so the above situations would cause the armature to skip across the data platters, scrambling little bits of data every time they hit.

That's much less of an issue today, but still not something to play with. Are you moving soon? It would be a good idea to make a backup of all the key files on your PC before you do. A battery backup unit/UPS unit is also a life saver when it comes to improper shutdowns, since it can help prevent those when your power blips or goes out entirely.

*If you'd like to know more about common causes of hard disk failure, there is a pretty good article on that here, and it covers some other topics like firmware corruption that I'm not going to touch on here.

Viruses are the very last way to lose your data that I am going to discuss. Certainly the average user should have antivirus software, anti-spyware software, and all the latest patches on their OS. This doesn't have to cost you an arm and a leg. AVG Free is free antivirus software that a lot of people swear by, and Spybot Search & Destroy and AdAware SE Personal are both free spyware prevention/removal tools that I love. Use all three if you're worried that you have a virus or spyware on your computer, because they all catch slightly different things. And remember that even then, they might not catch all issues on your computer. Browser Hijacks are notoriously hard to catch, for instance, but fortunately they generally don't pose a risk of data loss.


Solutions
Still reading? I hope so. Because now that we've covered the main risks, it's time to talk more about the various solutions available to you.

Hard Disk Redundancy in a single PC is an excellent way to protect yourself against all the various forms of hard disk failure, and it is generally convenient because it keeps your data on multiple hard drives in a single computer. It's unlikely in the extreme that two hard drives in the same computer will fail simultaneously for mechanical reasons. This obviously doesn't help at all when it comes to theft or fire, however, and probably won't be much use in the case of viruses or power surges either.

Simply copying data between two hard drives in one computer is a valid approach. You can do this by hand on any computer that has multiple hard drives (and if you only have one, it's generally very easy to install a second one).

If you have some extra money to burn, you can always set up a RAID mirroring array on your home PC. If you build your own computers, a lot of motherboards on both the Intel and AMD side have this built right in already these days--with the rise of SATA, these sort of controllers are much more prevalent than they ever were with IDE. However, there are advantages and disadvantages to RAID even if the cost isn't an issue, so it's something to consider.

Odds are fair that you could stand to lose a few hours' work if it actually came to that, so just copying your data between hard drives when you finish for the night might well be enough protection for most people (it is for me).

Data Recovery Software is a valid option if your hard drive is physically fine, but has had some sort of software-initiated data loss. For a virus to truly, permanently wipe your hard drive would take hours--and most viruses don't try to do this in the first place, because that would make them more fatal but less virulent (same as human diseases, they won't spread much if they immediately kill every host they infect).

If a virus was to go for your data, it would probably just try to corrupt or delete individual files. Corrupted files might be hard or impossible to recover from (this is why you should have antivirus software), but deleted files can often be recovered relatively easily if you act fast. If your hard drive is still working but you lost your data because you got a virus or accidentally deleted some files, software like File Rescue can get your files back much less expensively than sending it to a forensics service--but you must use this software as soon as you realize you have a problem, because your OS will overwrite deleted data eventually. This gradual overwriting process could take months or years if you never use much of your hard disk, but by the same token it could take minutes or seconds. Best not to tempt fate by waiting.

This software is also ideal when the FAT index on your hard drive gets corrupted. Essentially, all your files are in a central index on your hard disk, and when your computer "deletes" a file it just removes it from the index. Software like the above ignores the index and just looks for files on the main part of the disk itself. Improper shutdown of your PC can sometimes cause your FAT index to be corrupted, just as if a really nasty virus had attacked you, and so that's another situation where this sort of software comes into play.

CDRs / DVDRs are the most common way that home users choose to back up their data, but I don't think a whole lot of this approach. I've already mentioned how this sort of a backup is time consuming and something of a pain (thus making users less likely to actually do their backups), and I've also noted how these are just as vulnerable to fire unless you have them in a fireproof safe (and arguably, even that might not be enough).

But the other issue with CDs is that they don’t last as long as people thought they would. When CDs first came out, they were touted as likely to last 1,000 years, and CDRs up to 100 years. But these days people are already starting to have CDs (especially CDRWs, and to a lesser extent CDRs) that are failing because the plastic layers are coming slightly apart, allowing the thin sheet of metal between them to oxidize--thus losing your data. Gold CDRs are reputed to last longer than the regular CDRs (gold doesn't oxidize), but your mileage may vary. There just isn't enough of a history with CDs and DVDs to know what their life span will really be. Check back in a hundred years.

Other Removable Media besides CDs and DVDs can also be used. The most popular at present are USB flash drives, because they're just so darn easy to use (and these days they're pretty inexpensive, too). These are solid-state memory and don't require a battery to retain the data, which makes them ideal for semi-long-term storage. I'm not sure exactly what the range of expected life for these drives is, and I certainly wouldn't trust one as my only backup of anything (let alone my only copy), but as a quick, portable backup option this is a good one.

In the past, floppy disks and zip disks were the primary backup options for the home user. I used these myself, and in fact I still have some 15+ year-old floppy disks that still work fine and have all the data I put on them. But these disks have such a weakness to magnetism that their data seems less safe compared to any of the more modern alternatives. In a business environment I've also used the ever-popular tape backup setup, which can store massive amounts of data very cheaply, but I found the reliability to be too low for the purposes of my company and so we switched to the final solution I note below.

Remote Backup is the last solution that I'll touch on, and in my opinion it is by far the best. This solution has your data on another computer (or multiple computers) elsewhere in the world. That protects against any sort of hard drive failure, viruses, fire, computer theft--hell, it even protects against localized meteor strikes. Pick your non-global natural or unnatural disaster and this solution works, because no matter what happens to your home computer or laptop, your data is safely far away.

If you've got a large amount of data that you want to save in this way, you'll be hard pressed to beat the offering of Connected TLM (now a division of backup industry titan Iron Mountain). It'll cost you something every year, but you get a whole lot of redundancy. There are a lot of other solutions, as well, such as IBackup, but I know less about them. UPDATE: There is also now a program called Mozy, which I give more detailed information about here. This is now my far-and-away favorite Remote Backup solution.

One clever thing that you can do is email it to yourself with GMail or another free mail provider, so that you have a backup on their servers. I particularly note GMail not only because it is free, but because they give you so darn much space. Do note that if you have a POP account that automatically deletes off the server after processing messages, that kind of defeats the point--generally speaking your ISP's email services aren't going to work for this purpose.

Finally, if you have remote access to a computer in another location, you can just copy the files yourself. Have a friend or relative in another city? The two of you could arrange to swap files on some interval--either by setting up SSH on each computer (that's generally safer than FTP, which isn't usually encrypted), or just by emailing files to one another. It largely depends on how much you want to back up, file size allowances on your respective email providers (many have a 2MB or 10MB send/receive limit on individual emails), etc. If your employer doesn't mind, and you have a VPN connection to work, then you can also consider backing up a few key home files to your work computer. But please don't do this without your employer's knowledge and permission.

UPDATE: Free Online Document Management Systems are another way to go, but each one has its limitations. Google Documents allows you to upload as many files as you want, but they have to be HTML, plain text, MS Word, or MS Excel files. That's nice, but I also need to back up copies of my TreePad database. If you don't have any such files, then this might be an ideal solution for you. Just bear in mind that Google's terms of use for this service includes the line "Google has no responsibility or liability for the deletion or failure to store any Content." This is true of GMail as well, but it just bears pointing out that this doesn't have guaranteed retrieval like the paid services do. So it's a good idea not to put all your eggs in one such basket.

Yahoo! Briefcase is another such service, and it allows you to upload files of any sort, but there is a limit of 30 MB to what you can upload in total. You can pay to get more space, but you probably wouldn't need to until you had 20-30 full books stored on there. Their terms of service were not clear, but I'm fairly certain that they don't provide any retrieval guarantee as a true escrow service would, either.


Closing Thoughts
There are a lot of options here, and which ones you choose to take will depend on you and your own individual situation. Generally speaking it's best to use more than one approach to truly safeguard yourself. Just for the sake of example, here's what I do:

-I have three hard drives in my PC (for capacity reasons), and I manually copy my writing files between each one at the end of each writing session (I don't use RAID of any sort).
-I keep my desktop PC on a CyberPower CP425SL.
-I do have GMail, and very occasionally I'll send myself an email with a copy of my book.
-My employer allows me to, and so I back up my writing files to my work computer over VPN every few weeks. It would probably be better if I did this more often.
-Whenever I do work directly on my laptop, I copy that work to my PC as soon as possible.
-I don't use a cable lock with my laptop, but I also don't ever take if off my lap if I'm using out in public (which is rare in itself--mostly airports when I travel for work).
-I use both Spybot and AdAware to keep myself protected from spyware. This is a good idea in general.
-Trend Micro is actually the antivirus that I use, but it isn't free.
-I don't do any form of CD backup.

-UPDATE: I also now use Mozy.


If you've read all this and feel like this is just ridiculous over-protection and a complete waste of your time, consider this: you won't feel the same way if you actually do lose your data. Through my work at my day job, I've just seen too many computer failures to treat this cavalierly.

Outlining tools

In my last post, I talked a lot about the need for outlining. If you're the sort of writer (like me) who needs some sort of outline, you have a lot of options on how to do it. Until recently, I generally just wrote most of my plans using MS Word. That was fine when doing linear chapter plans, but less suitable when trying to map out central details about characters, settings, etc. Especially bad for quick checking between the characters/settings/data and the chapter plan.

To complicate matters, I was jotting down all sorts of things in a spiral notebook, too. That was convenient, because I could easily work outside in the sun on my back deck, or jot down notes from bed while I was half asleep. I often found myself coming up with a lot of ideas on the commute into the office, but I would quickly jot those down in an email to myself when I got to work, and then integrate those notes into my electronic files when I got home.

That approach worked all right during the course of writing THE GUARDIAN, but by the time I was ready to start working on the sequel, it was becoming unwieldy. This past weekend, I checked and I had five MS Word outlining-related files on my computer that contained over 45,000 words of content. Holy cow! I was introducing some new characters in the sequel, and having a really hard time cross-referencing them and their place in my story plans. Plus, there was still older content in that spiral notebook (50 pages front and back, and I write reasonably small). Something had to be done.

I started looking into some sort of software specifically for doing outlines. As a software designer, I've used numerous tools for planning out software architecture (Visio is my favorite), and I figured there had to be something of that sort that was more suitable for writing. At first, I mostly came up with a lot of old-as-the-hills programs, most dating from pre-95, and one nice-looking one that is only for the Mac (which I don't have). There were a couple of more recent ones that I found, but they were either expensive or strangely complicated (I'm trying to make this task easier on myself).

Finally, I stumbled upon what I felt were a couple of viable options. Here are some of the features common to many of the programs:
- Tree-based left sidebar for easy organization.
- Ability to add cross-reference links between one topic and another.
- Built in Rich Text Editor.
- The ability to link to external files (like documents or emails or whatever, if that's important to you).
-Compression that makes the outlining database take up much less space than the original MS Word files (30% less disk space, in many cases).
- Spell Check.
- Cross-topic search and replace.
- Topic bookmarks.
- Word Count.
- Mouse-based and keyboard-based commands.

Treepad
The first program is Treepad. It's very simple on the surface, it's cheap, it's fast, and I've ported all my MS Word files into it (A manual process, I must warn you. It took me some four hours to do this). This program has a wealth of options that was unmatched by any of the other programs I looked at. The formatting of most anything can be adjusted, including icons and colors, etc, for the tree nodes. This was also the only editor in which I noticed a tabs and indents control bar like in MS Word (you must turn on Advanced Mode for this to appear). It also has a handy little backup option that doesn't require you to shut off the program to make a copy of your database file.

On the downside, Treepad's word count feature only works in the current article you are looking at, and the ability to drag-and-drop sections from one location to another is a little buggy. The drag and drop works, but it takes me a several clicks sometimes. At least this isn't something that I need to do very often. Also, the spell checker is only available in their Business version or higher. That's only $20 more expensive than their Plus version, so I don't see this as really being a big issue. Some of the other programs still cost a little more than the Business version, anyway. Finally, topics can be cross-linked via making a literal link inside one topic to another (it's very easy), but there isn't a way to add a "virtual node" on the sidebar that is actually a link to another topic somewhere else. That's a feature I really liked in some of the other software.

ActionOutline
The second program, ActionOutline, is also pretty good. The interface is certainly prettier than Treepad, for what that's worth, and the program also has a better "feel" to it. It just felt better when I was clicking around and doing things. The drag-and-drop feature for moving topics around in here is really easy, which is nice. A lot of the features of Treepad are also found here.

However, it doesn't have any sort of a word count feature (and hey, I like to quantify my planning for whatever reason). Also, changing the icons of the nodes just isn't as easy. You can define all new node types, but then it takes extra clicks to be able to use them. I'd much rather just use the Insert key, and adjust the icons later at will if I need to. The Rich Text Editor here is beautiful to use, very easy, but a little simplistic. But even these things can be worked around pretty easily. The main killer for me is the lack of a spell check here. There also doesn't seem to be any way to add cross-reference links between different topics in the database, but honestly that's a feature I doubt I will be using much. The tree view is just too useful for me to bother with the cross-links yet.

WhizFolders
A third option is WhizFolders, which just didn't rub me the right way from the get-go. It has cool features like file linking, spell check, and even full word count support, but it's design is perhaps best described as "fidgety." It's always doing the not-quite-right thing as I click around and add topics and such (and I'm not one of those people who double-clicks everything. Their software is just set to do operations such as rename on way too short of a time interval with clicks). But, you can also do cross-links between topics using "virtual node" urls, which is a really nice touch (I must say). I'm not sure that I would use that feature much, since these outlines are really just for my own personal use (but I might in the future).

It's strange, because their software is a mix of really bad and really good design, in my opinion. The virtual nodes concept is awesome, and their Rich Text Editor is pretty darn good. But the way the windows are handled (one "listing" window has to be open in the background at all times, for instance) is just crazy. Give this program a try, and if you can look past it's quirks you might find you really like it a lot. I was very, very tempted, but ultimately decided to go with Treepad's much more solid-feeling design. Treepad still has more options in general, anyway.

Ultra Recall
The final good option that I found is Ultra Recall, which is really more for business users who want to keep track of what they are doing. It's got great journaling features and calendars and all sorts of file-linking options, but it's not what I would call a convenient outliner. It has loads and loads of features, but that's just not quite the sort of thing I found I was looking for. But if you're looking for an everything-organizer, this looks like it might be pretty good for that!

Conclusions
In the end, as I've already mentioned, I settled on Treepad. It had the right blend of features-to-sanity that I was looking for. I've sent them a note requesting that they fix the issues with the drag-and-drop, and asking if they would consider adding more comprehensive word count functionality. I haven't heard back yet (I sent it two days ago), but that's not entirely unsurprising. I'll let you know if I ever hear anything there.