SysAdmin1138 Expounds

10,000 hours

2008-11-29T15:05:00.002-08:00

I read an excerpt of a book a week or so ago. Always dangerous, as it lacks context. But the general principal of the book was the observation that to get really really good at something requires about 10,000 hours of practice. There are no 'naturals', just people who are naturally more pig-headed than others who can get to 10K hours.

10K hours is 2-4 hours a day for 10 years.

The studies were about things like child prodigies, or top tier athletes who get Olympic gold at age 22, and retire by 30. That sort of thing. It seems that almost all of these people started their thing by age 6, and by age 8 there was already a break between the kids who'd ultimately reach the peak of their field and those who'd merely be very good. The ones destined for peak were giving 2-3 hours a day at age 8, where the other group had cut back.

I believe this also applies to technical expertise. As anyone who has done any job searching in my field knows, there are real breaks for experience levels; 1-3 years, 4-6 years, 10+. Those of us in the 10+ area (and by now I am there with NetWare, and by the end of December I can claim that with Windows) are pretty much technical experts. We've put in the time over the years to get good.

However, we work in a field where, "Change or Die," is an accurate mantra. The IT industry of 2008 is markedly different than it was in 1998. Windows NT installs right now are laughed at. Very, very little of the operating systems and software in active use in 1998 is still able to be on a support contract. It is hard to be a 10K-hour expert in something in our field, you have to put in 8 hours a day for 5 years.

My first real exposure to NetWare was in a class I took for my CNA back in the Autumn of 1996. That was on NetWare 4.0, so at least my first experience was with NDS. In fact, my first job with NetWare was with 3.x, so I had to learn bindary on-the-job.

I consider myself to be an expert in NetWare. I've been actively administering it for 11 years now, so if I'm not across the 10K line I'm really close to it. This is only possible because the 'change or die' mantra has not applied to NetWare over the years. Lets take a look at the biggest disruptions of how things work in NetWare (kernel). This isn't incremental changes, this is fundamental re-learnings of how things work. Sort of like what all the Windows engineers had to go through when Active Directory came onto the scene.

The move to TCP/IP. This by far is the biggest disruption since 1996. NetWare 5.0(?) introduced the ability to do NCP over TCP/IP natively, and not tunneled IPX-over-IP. This required replacing IPX SAP, something the routers just did, with SLP, a service that needed configuration and setup.
The NSS file-system. This was a much lesser move than the TCP/IP one, as it worked on a general level (trustees, quotas, etc) the same as TFS did. Tweaking it for performance, however, was a dark art for many years and much learning was derived out of this.
Protected memory. A concept familiar to anyone who has used Windows or Linux, and all NetWare admins are by now administering one or both of these OS's. While some modules can't use it for whatever reason (iPrint, NetStorage) others (GroupWise) could.
Native File Access Pack. NetWare could do AFP since the NW3 days, the same for NFS. SMB was another story. It was with NetWare 5.1 that NFAP came on to the scene, and NetWare 6.0 where it came built in and performed much better. The ability to use protocols other than NCP for your Windows clients was embraced by many shops.

There were more changes, but in my mind these are the biggest four. You will note the complete lack of OES in this list. That's because this is a list of the changes to NetWare, and OES-Linux is not NetWare. OES-Linux represents the sort of "change everything you expect" that the rest of the industry does, that the Novell ecosystem hasn't had.

Over the last 12 years NetWare has remained markedly static. This has allowed enough time for people who don't do this every waking moment to achieve a high level of expertise with NetWare. While this is good for NetWare, it unfortunately shows how NetWare has lagged behind the rest of the industry.

It is my opinion that OES-Linux represents a decade of pent up change that needed to happen in NetWare but didn't. This is why old time NetWare admins are having such trouble moving to Linux, they're being asked to support an Operating System that they don't have anywhere near the same level of expertise in and that is uncomfortable. I know I'm moving from an OS that I know exceedingly well to one where there are still, "here be monsters," marked out on my mental map. I'm also having to give up, "10+ year experience with NetWare," in favor of, "2-4 years of experience with Linux," and that doesn't feel good professionally.

But... that is the nature of our field. Just when we get really good at something, it's time to throw it out and learn something new. That something may be an incremental change from what we know (Windows 2003 vs Windows 2000) or a complete break (NT Domains vs AD Tree). But, learn we must. Us NetWare wonks have just been sheltered from it for some time.

Budget 2009-11

2008-11-24T07:53:00.002-08:00

Sounds like the last analysis, the $5.1Bn number did not include, "caseload analysis." Or, analyzing increased case-loads to state agencies due to really bad economy. It sounds like when that is factored in, the projected deficit is closer to $5.8Bn.

Why is Washington State racking up the deficits? It's simple, we're more vulnerable to economic downturns than other states. That's because Washington has no Income tax, so most of the income for the State government comes from Sales Taxes. Sales Taxes are different from Value Added Taxes in that the taxed entity is the terminal consumer rather than the whole supply chain as a whole, so it is your average citizen that bears the full responsibility for the 8.3% tax on that new car. Consumer spending is projected to drop way off, and that in turn creates a major budget shortfall for the State government.

The Washington State sales tax is 6.5%, but Counties and Cities can add to that. Here in Bellingham, the sales tax rate is 7.8%, and down in downtown Seattle the rate is 9.0%. Counties and Cities also earn income through Property Taxes, which are less vulnerable to declining economic times (i.e. dropping house prices) due to how the taxes are calculated.

Here's hoping that the projections have been overly pessimistic.

Browser stats

2008-11-21T15:19:00.008-08:00

Some stats of who reads this blog (roughly):

FF3		45%
IE		42%
FF2		4%
Other		9%

And operating system:

XP		63.20%
Vista		14.98%
Linux		12.86%
OS-X		5.40%
Other		2.33%
Win2003		1.23%

Firefox out numbers IE, which is no surprise considering the type of search-traffic I get. I'm a bit surprised at how little Safari I see. Also interesting is that Linux edges out Mac on the OS chart, again not that surprising considering what I talk about here.

Budget numbers revised again

2008-11-20T12:13:00.001-08:00

To $5.1Bn. That number includes the $4.6Bn for the next biennium, but the $0.5Bn deficit for THIS year. Joy.

The revised budget is out

2008-11-19T12:34:00.002-08:00

As predicted, $4.6Bn instead of the $3.7 they were talking about earlier. What does that mean for me and Western? In the words of two of my bosses today:

For our state, revenues are currently projected to fall around 10% short of projected state expenditures. The magnitude of the shortfall is estimated to be about $4.6 billion. Since about half the state"s budget is protected from reductions through constitutional or other means, to make up the 10% shortfall would require an average of a 20% reduction in the rest of the state agency budgets. We are in that latter category of state agencies whose budgets, on average, would have to be reduced by 20%.

Just what does that mean for Western? The 20% reduction is in the 60% of our budget that comes from state support. So, it"s 12% of our operating budget.

So, expect a 12% cut to the budget. Right. Can't we just raise tuition?

If tuition were raised to the maximum currently allowed by law (7% per year) for resident undergraduates, the additional funds would reduce the magnitude of the cut to something in the neighborhood of 8.7% of our operating budget.

Still pretty hefty. However, the Legislature still has a full session to go over the budget, and it still has to be signed. Who knows what'll change between now and when Session ends. However, the prudent fiscal planner will be looking for cuts up to 12% of operating budget.

More close to home:

Second, [the President] has asked us to prepare scenarios about how we might respond to a 2%, 3.8%, and 5% permanent reduction for the next biennium.

So it isn't terribly likely that ITS, much less Technical Services, will be asked to whack 12% of operating budget. This is a good thing to this selfish person. 12% of operating budget for Tech Services would be people, since I'm pretty sure the sum total of our non-salary expenses is less than 10% of our budget. Whew.

Still, this is going to put a major lid on purchases for the next 3 years. Within that period our blade servers fall off warranty, which is going to incur certain replacement expenses (almost definitely for new ESX cluster nodes). On the other hand, it gives me a really good line to use for cold-calling vendors...

Spam drop-off continues

2008-11-18T13:14:00.003-08:00

It's been a week, and they haven't replaced their lost spam capacity yet.

The green bars are the 'clean' messages. But look at that! Over half of our incoming spam was from the same botnet/hosting provider. Wow. But, I do expect levels to go back to normal before too long.

Signs and portents

2008-11-17T22:15:00.002-08:00

Last Thursday I was over on download.novell.com looking for an eDirectory patch. I was staging up a new NetWare box and needed to see what the latest edir levels were. I knew 8.8.3 came out in August, and we're not there yet, so I needed 8.8.2 ftf2. However, I noticed that one of the searchable versions was 8.8.4. There was nothing in the list, but it was an option. It's not there right now, but it was then.

Thus emboldened I checked around a few more places. NetWare 6.5 SP8 was in the list, and still is right now. As is Open Enterprise Server 2 SP1. Both have the public betas posted, though.

But 8.8.4 was there. I saw it. Must have been a test or something. All this tells me that OES2 SP1 (a.k.a. NW65SP8) is just around the corner. Since we were told back at BrainShare that Sp1 would be in the Q4 time-frame, it's about due.

Bad budget predictions

2008-11-13T13:06:00.002-08:00

Quoth one news source:

Dunshee expects the state budget deficit will balloon to as much as $4.6 billion after next week’s revenue forecast.

You can guarantee that I'll be looking for next week's revenue forecast. $4.6 billion, for those keeping track, is a lot larger than the $2.7bn they were saying earlier. The measures I talked about earlier would have to be added to, if things really are as bad as that article lets on. The share of the $2.7bn WWU was allotted this year could be handled through use of reserve funds and other high finance things, or so said the President in September. A 70% larger deficit is another story, and that could lead to program cuts and/or lay-offs.

Lay-offs are a worry. I'm "professional" so there is no union behind me when the axe starts swinging, so positions like mine involve less paperwork and get more return in the case of termination. Unlike the Classified employees I don't have bump-back rights. In their case, if a, say, Fiscal Technician III gets terminated who previously held the title Fiscal Technician II, that FT-3 could bump-back into an FT-2. That bump-back is a displacement, so the FT-2 that got bumped would then exercise any bump-back rights she had, and on down the line until someone actually gets laid off. In my case, if they terminate me, I'm gone. Period. Much less paperwork.

But then, I don't know if that's how it works around here. The last really serious round of lay-offs was long enough ago that only two people in my department were around back then, and that's a lot of time for a management culture to change. However, if Technical Services gets handed an axe to apply to a position, my read of the lay of things is that I'm in the top 3. Also, we have no Classified employees in my department.

So, yeah. The budget forecast is something I'll really be paying attention to.

Very visible drop off in spam

2008-11-12T12:38:00.002-08:00

Thanks to this, we've seen a significant drop-off in spam. And what better way to show it than with a graph?

Dramatic, eh?

Let's see long it takes 'em to find a new host and re-tool. This is but a brief respite, but it's fun to look at.

NetStorage, WebDav, and Vista

2008-11-10T16:22:00.002-08:00

I figured out how to get it working! You need KB907306. This updates the Web Folders in Vista to support how Novell does WebDav through NetStorage.

In our case you'll also need to add the CA that serves the SSL certificate that's on top of NetStorage (a.k.a. MyFiles). But, it works.

No BrainShare for me

2008-11-10T10:42:00.003-08:00

Last week I asked my new boss if getting me to BrainShare was in the cards. I also threw out the alternative of the ATT Live sessions, which are fewer days and relatively cheaper than BrainShare. I also floated the possibility of me covering the plane expenses. As it happens Provo is about as expensive a city as Spokane, and Salt Lake City during BrainShare is equivalent to Bellevue. This is important since the travel ban is for 'out of state travel'. If I cover the plane tickets, that makes the cost the equivalent to having it in Spokane/Bellevue! Thus, I can go!

Unfortunately, not really. Quoth the new boss:

I highly doubt this'll work. Even Vice-Presidents are canceling travel plans.

If the grand high muckitymucks are honoring the travel ban, then the chances of me getting permission is next to zero. And is fully zero if I don't get buy off from my boss. Drat. Considering that the budget outlook for 2010 is even more grim than it is for 2009, the next probable BrainShare I can get to is 2011. Double drat.

Alarming error notices

2008-11-07T14:45:00.002-08:00

CommandView EVA can throw some alarming notices to the unwary.

Take this example.

Without knowing what's going on, the very first line of that is panic-inducing. However, this notice was generated after I clicked the "continue deletion" button in CV-EVA after we had a disk-failure scrozz a trio of RAID0 LUNs I had been using for pure testing purposes. So, while expected, it did cause a rush to my office to see what just went wrong.

Printer stats

2008-10-31T10:15:00.002-07:00

It has been a while since I last talked about printer statistics. Being a higher ed place of learning, we have student labs. These labs have printers. And like pretty much every place of learning I've ever spoken to, we do whatever we can to restrict printing to reasonable levels. In our case, we do this with PCounter.

For October, the sum total of pages printed (as of this morning, so it isn't ALL of October) from student printers we audit, was...

585,000 pages

That's a lot of paper.

Our busiest hour was 11:00-11:59, where 70,000 (12%) of those pages were printed.
Our second busiest hour was 13:00-13:59, where 63,000 (10%) of those pages were printed.
Our lightest hour was 4:00-4:59, where 360 of those pages were printed (the large majority from the HH154 printers).
The busiest single printer was one of the Library printers, with 48,000 pages.
The busiest lab (multiple printers) was the HH154 lab, with a combined total of 71,000 pages.
The busiest dorm printer was in the Fairhaven complex, with a total of 32,000 pages.
Of the around 22,000 active students we have, 10,509 (47.8%) of them printed at least one page from one of the audited printers.

The above comes to about 43 pages per full-time-equivalent student, or around 26 pages per active student for the month of October. The busiest user printed 892 pages in October, though only three users were over the 500 standard page-quota. Paper and toner are a significant cost of doing business.

The gift of security

2008-10-28T14:03:00.005-07:00

Last Christmas my parents bought me a 4GB IronKey. This is a nifty little device! And really, the gift of data security is rather thoughtful. And yesterday, it finally got the new firmware that enables Linux support!

Between then and now I haven't really been able to use it at work. And since transporting files between work and home is one of the nicer features of it, it has largely sat unused. But right this moment it is mounted to my openSUSE 10.3 workstation. This beats a floppy disk for transporting pgp/gpg keys.

Microsoft out-of-band patch, exploits released

2008-10-24T16:13:00.003-07:00

Earlier today, Bugtraq saw a couple of messages with links to actual exploit code for this patch. Now anyone can play!

On the up side, stuff built with this code will in all probability be detectable with IPS technologies. But that doesn't help devices in places that lack IPS, such as your local Starbucks.

An old theme made new

2008-10-22T12:42:00.002-07:00

Yesterday on Slashdot was a link to an article that sounds a lot like one I published two years ago tomorrow. The main point in the article is that due to the unrecoverable-read-error rate in your standard SATA drive (10^14 bits, or 12.5TB), and the ever increasing sizes of SATA drives means that Raid 5 arrays can get to 12.5TB pretty quickly. Heck, high-end home media servers chock full of HD content can get there very fast.

While it doesn't say this in the specs page for that new Seagate drive, if you look on page 18 of the accompanying manual you can see the "Nonrecoverable read error" rate of the same 10^14 as I talked about two years ago. So, no improvement in reliability. However.... For their enterprise-class "Savvio" drives, they list a "Nonrecoverable Read Error" rate of 10^16 (1 in 1.25PB), which is better than the 10^15 (125TB) they were doing two years ago on their FC disks. So clearly, enterprise users are juuuust fine for large RAID5 arrays.

As I said before, the people who are going to be bitten by this will be home media servers. Also, whiteboxed homebrew servers for small/medium businesses will be at risk. So those of you who have to justify buying the really expensive disks, when there are el-cheepo 1.5TB drives out there? You can use this!

Dorm printing

2008-10-20T09:11:00.003-07:00

On my post about finally running vista patrickbuller asked:

So you have printers that students in the dorms can print to? Wow. Do you audit all those and charge the numbers of pages against the student?

The answer to that is that we make big use of AND Technology's PCounter product. When paired with their PrintStations, it makes a very nice way to put a lid on unrestricted 'free' printing in the dorms. The PrintStations also make sure that only jobs people want to pick up get printed, which saves a serious amount of paper.

PCounter is core to our student printing. We'll only move our NDPS/iPrint infrastructure over to OES2-linux when Pcounter is supported on that platform, not before. We'll keep a 2 node NetWare cluster around just for printing if we have to. Since accounting support is one of the features that's supposed to be in OES2-SP1, it is my hope that PCounter will support OES2-Linux within a year after SP1's release. But I haven't heard any specifics.

Cool things with powershell

2008-10-17T13:25:00.002-07:00

Now that we're on Exchange 2007,we've had to figure out PowerShell. When I went to the Exch2007 class, it was pretty clear that Microsoft had redesigned their GUI tools under the 80/20 rule. 80 percent of the functionality that'll get used on a daily basis is in the GUI, and the 20 that gets used rarely or only by automation is on the command-line.

Which means that the once a year you go do something oddball you're hitting google to try and figure out the ruddy command-line options.

Any way, I digress. I've been writing a pair of powershell scripts to do some internal tasks (one of which is to create Resources the way we want them created), and have run into a few snags. The first snag is that a script that looks like this:

new-distributiongroup -Name $groupName -Type security -Yadda True
Add-ADpermission -Identity $resourceName -user $groupname -ExtendedRights "Send-as"

Won't work. That's because "new-distributiongroup" returns before the new distribution group can be acted upon by PowerShell. So I had to introduce a loop to make sure it was getable before I tried setting the permission. This is what vexed me. The loop I came up with is cludgy, but it does what I need it to.

$groupExists="False"
new-distributiongroup -Name $groupName -Type security -Yadda True
do {
   sleep -seconds 1
   $groupExists = get-ADpermission -Identity $groupName -blah blah |fw Isvalid
} while (!$groupExists)
Add-ADpermission -Identity $resourceName -user $groupname -ExtendedRights

While it works, when the script runs that loop creates a sea of StdErr output I don't care to know about. I'm waiting until it stops returning an error. Sometimes it takes only two seconds for the group to exist, other times it can take as long as 10. I still need to trap for it.

Today I finally figured out how to quash stderr so I don't see it. A very simple modification. It's in the test. Instead of "|fw IsValid", I use "2>1 |fs IsValid". This quashes StdErr, and still populates $groupExists. The script run looks a lot cleaner too.

The other thing I learned the hard way is that if you're doing multiple sets of mailbox or AD permissions, doing them too fast can cause the updates to collide. So I've taken to putting the above loop in to verify the previous permission mod has taken effect before I throw another one in. Annoying, but can be worked around.

Finally running Vista

2008-10-16T13:20:00.001-07:00

Why? So I can upload Vista32 drivers to the student NDPS/iPrint broker. No other reason.

It would seem that an amazingly high percentage of dorm-residents this year are running vista. Something close to 50%. So it has become rather urgent that we finally get Vista drivers into the iPrint enabled printers in their area.

Getting Vista64 drivers will be more challenging, as I don't have a V64 VM yet. :P

OES2 SP1 (public beta) has been posted

2008-10-15T09:33:00.003-07:00

The public beta of OES2 SP1 has been posted.

I believe the NDA has lifted, but I'm not 100% on that. Will check. But, some of the new stuff in SP1:

An AFP stack that doesn't suck. Or more specifically, an AFP stack that scales beyond 100 users and is eDirectory integrated.
A new CIFS stack written by Novell, so it can scale well past the Samba limit.
A migration toolkit in one UI, rather than a cluster of scripts.
A new version of iFolder
EDirectory integrated DNS/DHCP. But no eDir integrated SLP yet, open-source politics you know.
IIRC a beta of eDir 8.8 SP4.
The ability to put iPrint-for-Linux on NSS volumes (handy for Clustering).
And lots more I can't remember off the top of my head.

Go forth, and have fun. There is a beta-feedback box on the beta page I linked to above in case you find a bug and want to tell Novell about it.

One thing I think it is safe to say, is that even though it says "Beta4" on it, it's really a release-candidate. Only major bugs are getting quashed right now. UI freeze was a month or more ago, and strange, annoying behaviors may get "fixed in doc" rather than getting true fixes which will have to wait for SP2. Still report them anyway, since it'll go on the list to fix in the next SP.

Office

2008-10-14T14:38:00.003-07:00

OpenOffice has released version 3.0 in the last few days.

This naturally brought me to think about how uptake is going. WWU does offer OpenOffice as an option on our ATUS Lab machines, along side MS Office, but I have zero idea how often it is used by the student body. I have OO installed on my primary work machine, which is no surprise considering that's a Linux machine.

What is the future of OO? Will it ever supplant MS Office?

It seems to me that Microsoft is really pushing for SharePoint to take over from plain old file-serving. This is evident in the extent that SharePoint is integrated into Office 2007. I have to expect that the next version of Office will be even more tightly coupled to some web-based platform.

This is a problem for OO. While it may be possible to shim SharePoint integration into OO, perhaps through cunning uses of Mono, it means that OO will of necessity be one to three versions behind Microsoft in terms of features. Alternate platforms like Novell Teaming and Conferences are SharePoint-like, but they're not, you know, SharePoint.

Unfortunately, it looks like you can only go so far being able to read and create 100% MSO-compatible files. There's this other stuff that needs to be able to be done. WordPerfect learned this lesson in the years between MSO's ascension and the coming of age of OpenOffice as the, "Office package that is not Microsoft's," first choice. WordPerfect had the ability to save to PDF for years, and from what I hear is still the default choice in Law offices. However, WordPerfect is now the #3 behind OO and MSO.

OpenOffice has a difficult road ahead.

More on the budget

2008-10-09T11:31:00.001-07:00

The budget crunch I mentioned the other day has made the local paper. This is not that surprising, since we're one of the bigger employers in the area, and 20K students make a noticeable economic impact to the area.

Erm, about the budget

2008-10-07T21:24:00.001-07:00

From an email sent to all points from the U President this afternoon:

In the OFM spreadsheets received today, we were stunned to find that targets had been set for higher education. Western, today, is now expected, from the sorts of measures outlined in the August 4 memorandum, to "save" $1,827,000 in the current fiscal year. (This major reduction applies across all budgets, including instructional budgets.)

Add that to the earlier number, and our total budget reduction is NOT the $176,000 representing 1% of non-instructional budgets. It is $2,003,682.

Aaaaaand...

Further, we have been advised to expect these reductions to be permanent; that is, to also be a part of our 2009-11 budget.

Pardon me whilst I mutter things.

This means that it is nearly certain that we will NOT be getting any new hardware for the Novell cluster next summer. We'll have to do it on hardware we already own right now. This means I won't be able to partake of that lovely 64-bit goodness. Drat drat drat.

We're already under-funded for where we need to be, this won't help. Even with the storage arrays we just bought, in terms of total disk-space we've managed to fully commit all of it. There is no excess capacity. What's more, there is no easy way to ADD new capacity since any significant amounts will require purchasing new storage shelves.

In the intermediate term, this means that WWU will now descend into bureaucratic charge-back warfare. As service-providing departments like ours try to find ways to finance the needed growth, we'll start being hard-ass about charging for exceptional services. And they'll do it to us too. So if the College of Arts and Sciences comes to us and asks us for space to host 2TB of, say, NASA data, we'll have to bill them for it. And that cost will be a 'total cost' which will by necessity include the backup costs. In return, if we need 16 ethernet jacks added to the AC datacenter, Telecom may start billing us.

And I get a new boss Thursday. Happily, since there is overlap between outgoing and incoming they've been briefing a lot. This is to prepare the new guy for the challenges he'll face in his first few weeks flying solo. There may even be the odd phone-call for advice, we'll see.

Gonna get real interesting around here.

MSA performance in the new config

2008-10-02T15:54:00.003-07:00

Today I reconfigured the MSA1500 to run in Active/Active mode. While there, I also rearranged our disk arrays. We have 41, 500GB, 7.2K RPM drives in there. I created two, 20 disk Arrays, and filled each array with Raid 0+1 LUNs. This yielded 9TB of useful space. That extra drive will stay extra until we get an odd number of new drives.

Yes, a profligate waste of space but at least it'll be fast. It also had the added advantage of not needing to stripe in like Raid5 or Raid6 would have. This alone saved us close to two weeks flow time to get it back into service.

Another benefit to not using a parity RAID is that the MSA is no longer controller-CPU bound for I/O speeds. Right now I have a pair of writes, each effectively going to a separate controller, and the combined I/O is on the order of 100Mbs while controller CPU loads are under 80%. Also, more importantly, Average Command Latency is still in the 20-30ms range.

The limiting factor here appears to be how fast the controllers can commit I/O to the physical drives, rather than how fast the controllers can do parity-calcs. CPU not being saturated suggests this, but a "show perf physical" on the CLI shows the queue depth on individual drives:

The drives with a zero are associated with LUNs being served by the other controller, and thus not listed here. But a high queue depth is a good sign of I/O saturation on the actual drives themselves. This is encouraging to me, since it means we're finally, finally, after two years, getting the performance we need out of this device. We had to go to an active/active config with a non-parity RAID to do it, but we got it.

That darned iPhone

2008-09-29T12:18:00.002-07:00

The DHCP scope that is associated with our Wireless network filled this morning. We knew we were getting tight, and were in the process of getting a new system in place that could handle non-contiguous network segments as address pools. But that's not in yet.

The reason we ran out? Apple iPhones. When they come into contact with a Wifi network, they grab an IP address even if they're not actively using it. So. Full scope.

That isn't my department, so I'm not sure what exactly we're doing about it. But, it came up this morning,.