Let’s see if we make it past another 24 hours without something strange going on…

Through something of a comedy of errors, the server suffered some strange software issues that prevented it from working this weekend.  There was a bug in a recent update to the server software (running testing software) and that caused longer downtime than it should have due to various interactions between things on the server.  The good news is that this is mostly fixed.

Additionally, this downtime has taught me that there is yet still more to do in terms of getting the server able to stand back up on its own.  I’ve simplified the server’s setup a bit, and I have to write some scripts and other little glue here and there to tie down some of the things I’m doing so that the server can do things like go down and come back up without issues, all by itself.  Getting that done would be generally a good thing.  First things first, I have to figure out a decently reliable way to shutdown the system without having to do something like kill the containers and not give them the chance to cleanly shut down.  Ideally, there would be some sort of command that could be run on the system that would enable the containers to be shutdown.  This is slightly challenging, because you cannot just chroot into the directory tree that the VMs are running in and kill processes, because things like /proc inside the container aren’t visible to tools running on the host.  Oops.

So, I have some work left yet in terms of getting the server going the way it needs to be again.  After that, I’ll be working on it over the next couple of weekends to try to increase its robustness, and so planned weekend downtime for *.trausch.us and www.mischiefinoverdrive.us can be expected. 

• Bar a container from SSH’ing into its host system,
• Bar a container housing public services from SSH’ing into any of its sibling systems that are not also public (i.e., firewalling against the entire RFC1918 set of networks), and
• Bar a container housing completely private services from communicating with any non-RFC1918 addressed machine.

This way, I can ensure that services that I absolutely want kept private (such as the database servers) are that way without chance of mistake.  In situations where one of the private machines must have communication with the public Internet (for example, the database server needs to be able to replicate over the Internet to its sibling), a whitelist rule to permit exactly just that communication will be dropped in, and all is well and good.

My original plan was going to be to use KVM, and have the choice of running any operating system I wanted.  This didn’t work well with the storage strategy that I wanted to employ, though, without writing a lot of custom scripts to manage it for me.  Xen wouldn’t boot a Dom0 Linux kernel on my hardware (and NetBSD would not install successfully), and so I figured I had two reasonable choices left: FreeBSD jails and Linux Containers.  I installed FreeBSD, at first, thinking back to when I ran it on servers as my first choice of operating system.  The more I was thinking about the jobs that the server would be performing—along with the fact that the server was going to be performing many Linux-specific jobs, at least that is in the plans—I realized that jails under FreeBSD would probably not be the ideal solution.  So I started tinkering with container support on Linux.  So far, I am pleased with it.  It’s very much like FreeBSD jails, in that you can have a separate namespace for processes, mounts, and networking.  I think the mounts part is different from jails, since I don’t think you can mount new filesystems in a BSD jail (or at least, couldn’t at the time I used them, when they were very new; they may now have a tweakable or just support it out-right).  That said, it works nicely.  It can be easily managed with default system configuration files and minor additions to the init scripts.  (It can probably even be managed as an Upstart job, as I do with my DNS server, but I haven’t gotten there just yet.)

Currently, there are only two containers running on the server, though that will change very soon.  Containers/VMs were the very reason I needed to make this setup in the first place…

In Other News…

I am still stuck on AllTray.  I somehow still do not completely understand the code that I’m going to need to write to get the minimize-instead-of-close feature to work properly, nor how to elegantly handle various situations that it seems like will arise once it is written.  Once I “get it” (or someone else is able/willing to make it happen), the next release of AllTray will be made.  It looks as though that will definitely not be in time for inclusion in the forthcoming distribution releases for the latter half of the year, but I will try to build packages for then-current distributions and work to get the new AllTray into the next versions of distributions soon after that.  I’m not giving up; I’m just stuck at the moment.

I have missed the August 1st goal for releasing AllTray 0.7.4dev, mostly because I am trying to implement the last feature on the list of requirements for its release—the “intercept the close button” feature that everyone seems to think is a killer feature.  Unfortuantely, I have not yet found a clean solution that I am able to implement.  I have found a solution that looks like it will work, but the problem is that I cannot seem to figure out just how to make it work.  That is, I understand it in theory, but implementing it in practice is something I cannot seem to wrap my mind around.  This is unfortunate, and most likely means that 0.8.0 will be late (and thus not yet in Ubuntu), but it will be supported with the featureset it was published to have.  As a friend of mine told me, it is better to release late than to release an incomplete product.

I may go ahead and release what is currently trunk as 0.7.4dev, and bump the “close button” feature to a new milestone, 0.7.5dev, which I was not previously planning on having at all.  My original intention was to release 0.7.4dev, so that users could test it and provide feedback, then relatively quickly, advance to 0.8.0beta, 0.8.0rc, and then 0.8.0 itself.  However, because 0.8.0′s release will mean that I will not subsequently make any efforts to fix issues in the old AllTray series (as it will be the first version of AllTray since the old maintainer’s 0.70 release to be recommended for wide use and packaging in distributions), I am not going to release it in an incomplete state—so, late it will be.

In other news, the server that died some time back has had its replacement ordered.  The trausch.us domain will experience some downtime when things are migrated to the new server, which will hopefully be minimal—the IPv4 address block that we have is growing, as well, as we are in need of more IP addresses (and the cost of those is going to be subsidized, thankfully, as I honestly cannot afford them, even though I need them).  With any luck—and sufficient time—this means that some of the projects that I wanted to do that have not yet been possible, will be possible now.  Yay!

My primary reason for this is because it was a bit of a pain to work with BlogEngine.NET, and I simply could not work out all of the kinks.  As much work as WordPress is to maintain, it is at least easier than BlogEngine.NET.

I’m going to leave it at this for now.  I can’t find another system that I’d be willing to try out, and I don’t have the time to write one that I’d like (nor patch WordPress or BlogEngine.NET to fix what I’d like changed about them), so this will have to do for now.  Oh, well.

If there’s any issues with it that you notice, please let me know.

I also now have the ability to take the blog temporarily off-line so that I can make broad changes when I need to without worrying about temporarily breaking one of the scripts or something. That is always a good thing, even if I never use it.

As I mentioned in the last post, I now use reCAPTCHA here. I did this simply because, well, I was getting a lot of comment spam and using the moderation queue to fend it off was really just not working.

There are some minor changes that I still need to make. If anyone linked to my categories in the past, or subscribed to category feeds, you will need to do so again because with this upgrade, WordPress introduced the concept of “tags”, alongside categories. So, if you want to subscribe to what I was calling tags before, they are now categories here. If you want to subscribe to tags, well, feel free to do so, but they are bound to be far more specific than categories.

Anyway, there are probably other, little, gotchas here and there. If there are any major issues, let me know by dropping me a line here. Also, I do know that some old posts do not format correctly with this new theme—I am not going to worry terribly much about it. Another thing is that I do not have Internet Explorer, and so if there are issues with this new theme and Internet Explorer, I am going to have to be told about them. Side note on that, though: If it is older than Internet Explorer 7, I am not worrying about it, so upgrade to IE 7 and see if any issues persist. If they do, leave a comment with a link to a screenshot, or email me a screenshot so that I can see what the problem is and hopefully fix it.

I have also made it a bit easier to actually comment here. I had to re-write the OpenID part into this new theme, but that’s alright. Also, I have added reCAPTCHA support for the comments, and removed the moderation queue altogether for 99% of cases. If spam continues to be a problem, then I will probably re-enable the mod queue, but I don’t think that is going to be an issue.

After they’re done migrating, the trausch.us sites (including mine here) will be back up and running, barring any issues with the migration (so here’s crossing my fingers).

Good luck, NFSN!