Justice Dept. Official: We Could Get Lois Lerner's Emails From Backups, But It's Too Hard So Naaaaaah
from the facepalm dept
I try not to go for conspiracy theories generally, but this ongoing IRS nonsense involving conveniently disappearing emails potentially pertaining to the scandal involving targeting certain groups is making my skeptics beacon go off. The official story essentially involves a computer (server?) crash that obliterated the email data of several email accounts that would otherwise be of great interest to those trying to figure out who in the Obama administration knew what about how the IRS was operating. That crash somehow also involves the destruction of any local backups these IRS folks are required to keep as part of their job.
But what I imagine pretty much everyone that has even a modicum of interest in office technology is asking is what about the kind of backups that are typically run for disaster recovery and offsite purposes. You know, like tape drives (blech), D2D backups, or remote storage. It’s difficult to believe that all copies of the data for those email accounts had somehow been disappeared by accident. Or, really, disappeared entirely at all.
As it turns out, there is backup data for those email accounts, but if you’re thinking that we’re finally going to get to the bottom of this nonsense, you’re a silly naive person that hasn’t witnessed our glorious government at work.
“A Department of Justice attorney told a Judicial Watch attorney on Friday that it turns out the federal government backs up all computer records in case something terrible happens in Washington and there’s a catastrophe, so the government can continue operating,” Judicial Watch president Tom Fitton told Fox News’s Shannon Bream. “But it would be too hard to go get Lois Lerner’s e-mails from that backup system.”
Now, I know, Fox News and blah blah blah, but this is a claim worth paying attention to mostly because it only makes sense to begin with. Well, the part about there being backups, I mean. The suggestion that such backups are so difficult to parse and recover that the administration isn’t going to get them for review makes zero sense because that’s what damned backups are for.
Think about the logic here for a moment. The reason the federal government has backups of data in place is because there’s a great deal of horrible crap that could happen to their facilities. Natural disasters, terrorist attacks, hallmark-level incompetence, Area 51 aliens going Rambo on the computers. These are the things they have to prepare for and they need the ability to restore data from a remote location should one of their facilities fall to our alien overlords. To say that restoring that data is too hard to do when requested is to say that they’re ill-prepared for a disaster. The suggestion that the government can’t retrieve emails for court or congressional review is to say that they wouldn’t even meet FINRA compliance, something that itsy-bitsy little trading firms are required to meet.
Fitton said his group plans to ask a federal judge to order the IRS to hand over the e-mails, which conservative opponents of Lerner want to see in order to determine if there is a link between President Obama’s team and the IRS’s targeting of tea-party groups.
“If this backup system is working, Lois Lerner’s e-mails are there,” Fitton said.
And if the backup system isn’t working, then fire everyone everywhere because this kind of thing is no joke. I’d say it’d be a bigger scandal to have any portion of the federal government not have basic backup and DR systems in place than to find out that any portion of the Executive administration was encouraging the IRS to target certain special interest groups.
Filed Under: backups, doj, email, irs, justice department, lois lerner
Comments on “Justice Dept. Official: We Could Get Lois Lerner's Emails From Backups, But It's Too Hard So Naaaaaah”
Mike, you underestimate the Government. From my experience there are places where these e-mails WILL vanish if the account holder doesn’t have a personal, local backup. Cross your fingers and hope there’s no fire in the place.
Re: Re:
Mike?
Re: Wet Socks
Ninja do me a favor and look at the author of the article and after you realize it’s Timothy who wrote this one not Mike please go hit your self in the face with a pair of wet socks to knock the derp out of yourself.
Re: Re: Wet Socks
Upon information and belief, Ninja is channeling a composite of the trolls, tin-hat fanciers and others on this site who seem to lack the sort of mental granularity necessary to distinguish the difference among the various authors. It’s come up before.
Re: Re: Wet Socks
I hope I’m not the only one here to notice that Masnick and Geigner have very different writing styles.
Re: Re: Re: Wet Socks
I hope I’m not the only one here to notice that Masnick and Geigner have very different writing styles.
The difference is usually clear before the end of the first sentence.
Re: Re:
It’s beyond me that whichever judge is overseeing this mess hasn’t pulled the government’s lawyers into court and told them in absolutely plain and simple terms that if they don’t manage to find and deliver the relevant documents to the court in a reasonable amount of time, someone important is going to find his ass in a jail cell until they are so delivered.
Oh, the stories I could tell you about backups
Fortune 100 corporation: no offsite backups.
Major university: backups readable by everyone on the network, i.e., Alice could retrieve Bob’s files.
Another Fortune 100 corporation: backups weren’t readable. Guess how they found out.
And on and on and on. Most organizations are quite happy to buy overpriced overblown overhyped horribly insecure backup “solutions” from greedy vendors, deploy them, and mark the problem as “solved”. Few — precious few — are actually willing to engineer solid solutions based on well-tested well-documented code (e.g., dump, tar, rsync and other basic tools) and then actually perform ongoing monitoring (as in “read the logs”) and testing (as in “see if restoration is possible”).
The overall state of backups across the world is best classified as “miserable”.
Re: Oh, the stories I could tell you about backups
The overall state of backups across the world is best classified as “miserable”.
As an administrator of a network professionally, and as a volunteer tech support person on the side (who in this business doesn’t hold that title,) I wholly agree.
However, that being said, it is amazing the power of TAR/DAR (or even 7z, for those Windoze aficionados,) and external USB hard drives. They’ve saved me from some really serious stuff, just as long as you write to multiple hard drives, and periodically test your backups on a non-production system.
Still miserable ghetto-tech, but it would have helped here, if it wasn’t in the best interest of those involved (allegedly) that it not work.
Re: Re: Oh, the stories I could tell you about backups
I don’t know if I’d classify it as “ghetto tech”. I work with highly sensitive data (PHI) and while we do daily backups of all our servers to a centralized backup repository, we also do periodic full backups of everything and stash those on encrypted compressed external hard drives which are kept off-site.
And yes, we test them. 😉
That’s why in all the years I’ve been here our success rate at restoring files is 100.00%. And why, if the building burned to the ground tomorrow, I’m confident that we could restore our operation to the state that it was in as of the beginning of the month. (Which, for what we do, is acceptable.)
But I’m painfully well aware that we’re one of the few exceptions in a sea of mediocrity and incompetence.
Re: Oh, the stories I could tell you about backups
Which is why:
Having “backup” just to check a box that says “yes we backup our systems” means nothing if the process isn’t tested. Disaster Recovery is a level above basic backup, but for critical systems (and can you reasonably claim that the IRS isn’t critical?) it has to be taken seriously. Fire the IT department; if they can’t do it in-house, outsource it; there are companies that do this for a living.
Re: Re: Oh, the stories I could tell you about backups
Or, you know, maybe give them the budget and man-hours that they need to do the job right (which is the most likely culprit when you have IT issues like this).
Re: Re: Re: Oh, the stories I could tell you about backups
… point. There’s almost certainly been skimping on safeties and backups to stretch inadequate budget(it’s rare that an IT department gets an adequate budget, it seems to me – particularly for things that don’t demonstrate a benefit unless something bad happens).
Re: Oh, the stories I could tell you about backups
just as an aside, we have a semi-strict backup regimen where i work, but i’ll be fucked if i understand what good it does us…
on a couple of occasions, i have gone to them about ‘oh, that subdirectory/file is missing, can we restore from backup?’, or ‘i need a previous version of this file, can we restore from backup?’, and the answer is ‘no, we can not’… ? ? ?
apparently, the ONLY thing our ‘backup’ is good for, is if the whole building burns down, they can recreate the server’s files…
great:
1. realistically WHAT are the odds of that ?
2. 99% of the time, it is NOT a TOTAL disaster we are recovering from, but small disasters here and there…
3. IF you can’t get a previous file from your ‘backup’ system, it is not a backup system, but insurance…
4. the daily backups are then MOSTLY, MOST of the time, a complete waste of time & resources; we could back up once a month or once a year for the sake of the server (data files are another matter, but we aren’t able to get to those anyway, so WTF…)
shrugs beyond my pay grade…
Re: Oh, the stories I could tell you about backups
I can agree with and expand upon this.
I work for a (non-US) government department.
Part of our IT, unfortunately including backup responsibility, was (not any longer) outsourced to a 3rd party outsourcer, one of the big ones.
I once had occasion to view the contract, specifically that dealt with backup requirements, with said outsourcer. Nowhere, not once, did it include the word “restore” in it. Or any phrases or concepts that could correspond to the term/concept, restore.
Therefore, in theory, they could backup to /dev/null (wow, backups run so fast now, use so little CPU or network bandwidth…), as nowhere was there ever a requirement to be able to restore the backups. Sometimes they could restore data, often they couldn’t. When they couldn’t, they’d just shrug their shoulders and say it can’t be restored, not their problem.
If too hard to restore then they are useless
Your backups must be tested
So you know they work as expected
Offline is best
So you can rest
When lightening strikes unexpected
Re: If too hard to restore then they are useless
If you don’t test your backups… you don’t have backups!
Re: Re: If too hard to restore then they are useless
In Russian Ukraine…
Backups Test YOU!
And Steve Stockman is subpoenaing the NSA to see if they have access to the emails or related metadata:
http://stockman.house.gov/media-center/press-releases/stockman-requests-subpoena-of-nsa-s-white-house-irs-phone-logs
TL,DR; Stockman and his ilk only oppose NSA surveillance when it benefits them personally to do so. When it incriminates someone they don’t like (Fruit of the Poison Tree, Anyone?) they’re all for it!
Re: Re:
Said every politician ever whist the sheeple look on it fake disgust.
I am beginning to think the Americans only hate the Elected because they are shit faced jealous of them. They don’t seem to get getting them out.
Can we just execute whoever slipped in the pork that was kicked back to the provider?
If it is to hard to get the data back, the ENTIRE thing is a waste and we should file suit. Anyone involved in continuing the contract should be assumed to be at best incompetent at worst bought off.
Is it time that they stop digging into our data & lives and maybe consider they should look in a mirror and fix the problems there.
First talking head that tries to lay this at the presidents feet gets shot.
This is decades of corruption and stupidity coming to light, and they need to find the will to fix it without finding a way to line their own pockets on the deal.
Re: Re:
Until the critters in congress start saying the words understand that they are collusive in the problem.
There is a big reason that they do not call each other out for actual corruption… they are all corrupt! That is off limits!
sounds very similar to the piss poor excuses used by the City of London Police when asked via FOIA for info on contacts between them and the entertainment industries.
Well If they can’t manage to do their jobs maybe they need replaced and jailed for obstruction of justice, When you I can be fined by the MPAA/RIAA into and beyond poverty for sharing files , Am I being unreasonable to ask for these guys to get the same treatment or worse for this offense .
Come on, really. A server crashed that took out the mail only of those of interest? Really? Lois Lerner’s e-mails have been lost as a matter of policy, not fact.
If it ever becomes more useful to have those, they will magically be found never to have been lost at all.
A moot point?
Of what use are these emails when it is a well known fact that the IRS investigated applicants from all political persuasions with somewhat equal vigor? In addition, it has been reported that the only organization denied their coveted status was of a liberal slant. Add to this, that the status is not even needed in order to initially claim the exemption.
It is a side show, like Benghazi, trumped up pile on bullshit mountain.
Re: Re:
It is a side show, like Benghazi, trumped up pile on bullshit mountain.
This particular story is not about the merits of the investigation, but the IRS response to it. The target of a Congressional investigation cannot be the party that decides whether it’s legitimate.
Re: Re:
Well, if it’s such a moot point, then why all the lying & stalling about the emails, simply release them and shut up the critics…case solved..let them prove it is nothing but a witch hunt…make them look bad…if they can…course..if the emails back up those claims…then what..it’s a plant by the extreme right I guess..
Purposeful
This is totally purposeful. As an IT professional for most of my life, there is no way that data wasn’t backed up and no way that the backups are saved SOMEWHERE – normally – ESPECIALLY for the IRS.
You can be DARN SURE that if those were emails that had to do with auditing some poor sap and ruining his life that they would be found INSTANTLY.
The IRS is the largest, most well-funded and most brutal terrorist organization in the world. Never forget that.
Re: Purposeful
This does seem most likely to intentional. However…
“there is no way that data wasn’t backed up and no way that the backups are saved SOMEWHERE – normally – ESPECIALLY for the IRS.”
I think that’s a step too far. Given that tons of companies — including major corporations and tech companies — totally fail in terms of backing up their stuff, and we’re talking about the IRS here (who has blown through literally billions of dollars in failed attempts to modernize their computer systems), the idea that their backup systems don’t work adequately is not beyond the realm of possibility.
Re: Re: Purposeful
Care less about all of the failing to backup their shit fucksticks.
I work in IT, the politics and not the technology is the largest barrier.
There should only be 1 rule in the political world. If you cannot produce the records you say you keep, then you must fall upon your sword! There is simply no excuse for any of this!
Well, it’s easy to see how eager and willing they are to get to the bottom of this whole mess, that’s for sure. /s
I think we need a complete breakdown of how taxpayer money is being spent , How about an audit of the IRS.
Mail server blues
The only explanation that has ever made any sense to me (I’ve done tech support for almost 20 years) is this.
I’ve read in the news that the IRS uses a Microsoft Exchange system for email. Each user gets X amount of data storage on the server that is backed up. The user goes over that limit of X. There are two solutions to this in general. The first is increase the size of the storage on the server and the back system. The second is to set up a personal storage file (.pst) on the local computer in that person’s office. The user (with the help of a tech, usually) chooses what emails are moved from the server to the local file. The office systems are not usually backed up. If the second option was deployed by the user or tech staff and later on that office system’s hard drive crashed… well, then the data could be gone for good.
Now, they say that there is a backup? If so, I have no idea how they’ve set up this system.
Re: Mail server blues
Even if the email were removed from the server and stored locally via a .pst archive, the emails would still exist on backups that were created in the time-frame that those emails were on the server. They may have to look through a lot of old tapes, but that is their fault for not storing the .pst in another location on the network that also gets backed up.
This should be a lesson to anyone who isn’t tech-savvy in today’s world to learn the basics of how these things work so they can filter out the bullshit these people are feeding us.
Re: Mail server blues
“The user (with the help of a tech, usually) chooses what emails are moved from the server to the local file. The office systems are not usually backed up. If the second option was deployed by the user or tech staff and later on that office system’s hard drive crashed… well, then the data could be gone for good.”
That’s not typically how email backups work, in my experience. Usually, any competent organization is going to be doing some flavor of email archiving, in which a journaled Exchange account funnels copies of all email traffic to the archiving system, which is separate from the Exchange box/cluster. The user CANNOT control what is backed up by design. No system which allowed that would meet the many compliance standards the government and SGOs have set up for various industries.
As a device on the network, that message archiver is also backed up, along with the Exchange box/cluster. Backups of any legitimacy involve a local and at least one offsite location. This is harder with tape drives, which is why tape drives should go away and die forever, but there should always be at least 3-4 geographic points from where emails can reside. For all of those to disappear isn’t believable.
Re: Re: Mail server blues
Usually, any competent organization
I think I see your error. 😉
Re: Re: Mail server blues
According to what I’ve read (over at Ars Technica and linked from there), the problems appear to be:
The tape backup drive that the emails were probably stored on, (aside from those that were lost on the computer itself), was recycled as part of a rolling back-up system – (probably every 6 months).
If you don’t like that, blame Congress, not the IRS, for not providing funding and support for a proper archival system as opposed to a short-term backup system, which they currently have – though since, of course, government funding for anything other than the military is currently being cut, YMMV.
They MIGHT be able to restore some data from the recycled back-up tape, but don’t put any money on it…
This is NOT the fault of the IRS at all – which a lot of people won’t want or like, so don’t expect any of this to become part of the main narrative…
Re: Re: Re: Mail server blues
They are REQUIRED BY LAW to maintain these records longer than this plan supports, so I actually find the idea that they simply re-use media because they do not have proper funding dubious.
What really gets me is that this is the IRS – the same organization that requires me to maintain tax information for SEVEN YEARS. If they can claim “lost by stupidity” and be just fine, why can’t the rest of us?
Re: Re: Re: Mail server blues
“If you don’t like that, blame Congress, not the IRS, for not providing funding and support for a proper archival system as opposed to a short-term backup system”
No, this problem is squarely on the shoulders of the IRS, not Congress. Congress has given them PLENTY of money to fix these issues — on the order of billions of dollars.
The IRS blew through all that money in a couple of projects that ended in failure.
“And if the backup system isn’t working, then fire everyone..” When was the last time there was a firing for incompetence within the government? If we did that, all we would have is very clean offices with no one in them.
Because the janitors would be the only ones working..
Re: Re:
That could be funny, we’d get to watch lobbyists try and bribe janitors to clean the offices of the ex-politicians they were trying to influence ‘extra thoroughly’.
Re: Re:
Sounds like a plan.
Re: Re:
im good with that.
Typical and Far Too Common
I am an engineer employed by the federal government. Permit me to explain to you the convoluted nature of how I have to back up my data on my issued machine.
In order to make a backup, I have to plug an external drive into my machine and manually run my backup software, rather than having an automated backup to the network. I start a obscure piece of software that I have been directed to use to make backups. I then start the process, which basically locks me out of actually using my machine until the backup is complete. There is a strong encryption requirement on any data put onto an external drive, which means that making a backup is nearly an all-day affair to allow my (somewhat underpowered) machine to crank through all the encryption. I can’t leave the software to run unattended, though, because it is not quite smart enough to handle that. I can finally use my machine again once the whole process is completed.
I usually line up some work to do in the lab or on my other computers when I intend to run a backup on my primary machine. Otherwise, the day tends to drag.
I had to jump through hoops just to be able to do this. You see, I was not issued the external drive. Most of the machines are set up to *not* allow backups to external drives. If I did not have an external drive, I would have to use optical media. We also only recently (within the last two years) received DVD burners as standard in our machines. I suppose they think we are just supposed to back up to an assload of CDs, like anyone will do that with any regularity.
The trouble is, the backup and restore process should be easy. That’s how you get people to actually make the backups they’re supposed to! That’s how you engineer things so that the records aren’t lost when a hard drive dies. However, making the process easy would make sense and be efficient, so there is little chance of it happening.
On another note, I am not convinced that the emergency backup alluded to will be of any use. If the IRS system is set up similar to mine, then they probably only have 1995 levels of email storage (25MB or so) on the server that would be backed up at any one time. Either that or the files would mysteriously disappear before they were restored.
Re: Typical and Far Too Common
Very good points. However, you state that you had to “jump through hoops” to get this system set up. I cannot imagine a high level executive of any company allowing their computer to be locked out of their systems for a full day or sit and hit “enter” at regular intervals. You’re right. Encryption and backup should be easy for the end user but you state that it’s not in your experience in the federal government. In my mind, you’re strengthening the argument that there is likely no backup of the local office system.
As for going into older tapes to get the data. Yes, that is entirely possible. However, if those tapes were being recycled and reused (common practice in a LOT of businesses) then the data could have been destroyed at any point.
Re: Re: Typical and Far Too Common
That was my point exactly. Unless you are technically savvy enough to both understand the need and get past all the roadblocks, backups of the local machine are likely done rarely, if ever. It’s a problem when you need to be accountable to the american taxpayers.
Re: Typical and Far Too Common
That’s pretty insane.
However, these are emails we’re talking about. The backups for them would not be off o your machine, but the exchange server. Surely, they aren’t backing up that server using USB drives.
Everyone who thinks there are backups are wrong
This subject has been covered in depth, so I don’t understand how misinformation like this goes around.
Commenter Chad Holbrook hit the nail on the head. It has already been well established that Lerner saved her email to a PST file on her local computer because their Exchange mailbox size limit is so low. The hard drive on her workstation crashed, was disposed of, and was replaced. Her workstation was NOT being backed up.
Yes, those emails were on the Exchange server for a period of time, but backup tapes get recycled and overwritten.
Those emails are irretrievably lost.
Re: Everyone who thinks there are backups are wrong
Oh, and to the folks who say the PST files should be on a network share so they would be backed up: good idea, but no dice. Microsoft specifically says PST files over a network link is a no-no.
Re: Re: Everyone who thinks there are backups are wrong
So how come my .pst files work fine onto a network drive?
Oh noes, my .pst files just vanished in a puff of logic!
Re: Everyone who thinks there are backups are wrong
Those emails are irretrievably lost.
Except now the IRS says they’re backed up, but too hard to get to.
Re: Everyone who thinks there are backups are wrong
If they were lost, they’d just say so, but they haven’t, instead they’re just ‘difficult to get to’, which suggests they could recover the emails, they just don’t want to bother with trying.
Re: Everyone who thinks there are backups are wrong
I think we aren’t misunderstanding. We’re just giving the IRS a well-deserved hazing because their backup system is clearly inadequate. No, that’s not harsh enough. It’s incompetent.
Backups. We don't need no stinkin' backups.
I used to work for a company you all have heard of. (Check my LinkedIn profile if you really care which one.) Back in the day (October 1989) our Bay Area office was destroyed by an earthquake you may also have heard about. Live and backup data within the office was destroyed. The policy at the time (now policy is very different) was not to rely on any local offsite backup facility as that could just as easily be destroyed by a quake. Rather, the office had a deal with a taxi driver to come every evening after daily backups and carry around a copy in the trunk of his cab. Next night he’d be back to swap disk packs and would carry the new copy.
The office recovered from the quake using data from the disk packs in his cab.
As to Tim’s post: I’m a Democrat who has supported Obama from early on (though break with him on several issues of TechDirt focus). Despite my general support, I agree with everything Tim wrote–something does not smell right here. They do need to recover those emails no matter what is on them.
Re: Backups. We don't need no stinkin' backups.
You know – a really obvious location for backup data is a reliable cloud hoster. Pity Hollywood is putting so much effort into destroying all of those because ** piracy **
Re: Re: Backups. We don't need no stinkin' backups.
There are a TON of good local/offsite D2D backup solutions. At my 8-5 job, we have used a couple of them to power a managed BDR service. We’ve switched them around a couple of times, so this isn’t to pimp the Digerati Group service, but they’ve included the Barracuda Backup solution, Datto, and Veeam for virtualized environments. They all include onsite/offsite backups and can work with a message archiver for journaled Exchange accounts.
This stuff is waaaaaay more simple than people are led to believe. The idea of data residing primarily on a local client instead of a server and having the responsibility for backups foisted upon common users is laughable to the extreme….
Re: Re: Re: Backups. We don't need no stinkin' backups.
Don’t tell Hollywood…
Re: Re: Re: Backups. We don't need no stinkin' backups.
Veeam is awesome and far easier then you would expect. Took me 30 minutes to install and setup and 15 minutes to test after the first backup cycle.
Backups & DR
I used to work in IT for banks. We maintained a 3-day rotation on our DR backups, and tested the business recovery process every year. Understand, however, that the DR system is designed to recover the business to YESTERDAY’s work. If it was deleted last week, it won’t be on yesterday’s backup, and likely won’t be needed to recover the business.
Too hard?
Heck, I’ll do it for them. Dear IRS, I volunteer. I’ll do it pro bono.
Re: Re:
Heck, I’ll do it for them. Dear IRS, I volunteer. I’ll do it pro bono.
As far as I am aware, unless things have changed, it is very unlikely for the government to accept your volunteer/pro-bono work. There are some places that might take your Volunteer work, but they aren’t many. The government assumes a liability when dealing with volunteers, and being risk adverse, they tend not to like it.
Of course, you could go to work for a contractor, which means someone else gets paid for your volunteer work. But that would require the standard government contracting process, which would likely make it not worth your time or effort.
pst’s over a network link tend to corrupt very easily, I know from experience.
Here’s the problem, they are too busy archiving *the people* of America’s emails so when they go looking for “Lois Lerner” it returned multiple accounts and they don’t want to sift though them all.
Love the disparaging remarks about Fox news
Love how you diparage Fox news. Are you implying the Communist News Network is somehow better? Anyone who doesn’t watch Fox is basically getting the same story from 100 places. Fox is the only one showing the “other side”. But hey, don’t take your blinders off, feel free to vote for Hillary. Of course she couldn’t even run the State Department or keep Obama’s Benghazi lie going for more than 2 days. But if it weren’t for Fox news, you would still beleive the Benghazi lie too.
Re: Love the disparaging remarks about Fox news
Listen, idiot. I specifically put that line in to COMBAT people dismissing this simply because it was on Fox News. I did the opposite of what you’re claiming.
Don’t forget breathe, now….
Re: Love the disparaging remarks about Fox news
“Love how you diparage Fox news. Are you implying the Communist News Network is somehow better?”
That’s a false dichotomy. Just because FNC is abominable doesn’t imply that CNN is not. Television news generally tends to be truly awful.
“Anyone who doesn’t watch Fox is basically getting the same story from 100 places.”
This is demonstrably untrue. I don’t watch FNC, but I’m certainly not getting the same story from all of the news sources I do pay attention to.
“Fox is the only one showing the “other side”.
You think there’s only two sides to any given news story? If so, then I think that you need to check your own blinders before pointing out the ones on other people.
Re: Re: Love the disparaging remarks about Fox news
And if scientists or Democrats said the moon is made of rock and gravity makes you fall, Fox would find ‘experts’ that disputed these facts… as its ‘opposite opinion’.
Disaster recovery backups do not equal daily, weekly, monthly, rotating backups.
Sometimes the software used is even different – imagine that.
One scenario would be to do machine backups periodically. These machine images have to be restored in toto (as opposed to file by file). If that machine is part of a complex infrastructure, the corresponding infrastructure also has to be restored in order for that machine to function.
For example, I do yearly DR tests on a small environment. That environment requires 12 virtual systems and two networks to set up. During a DR test, those new machines and networks are provisioned, then the DR information is loaded in a documented order. Finally, both system and user tests are run to complete the drill. Even with a small environment such as this, it takes 6 hours from start to finish.
DR backups recover what’s required for business continuity. This may NOT be everything. If everything goes as planned, you recover from a DR snapshot, then use off-site backups to bring the DR snapshot up to date. If everything works as planned, you lose 1 day’s worth of information. Yes, you can do incremental backups more frequently (mirrored via a MAN) if your business requirements dictate it.
Now, imagine this scenario. Backups have failed, so all you have are DR snapshots. If you have the DR snapshot that contains the mail in question, you’re in luck. You still have to set up enough of the infrastructure to bring up the environment from the DR snapshot. If you don’t have a DR snapshot that contains the mail in question (depending on full or incremental backups to completely restore the environment), then you’re out of luck.
Still, it can be a major undertaking to recover single pieces of information from a DR system.
We can solve this...
Requestor: Oh. Too hard to get it off the backup. That makes sense I guess. But that’s no problem. Just give us the whole backup and we’ll get it off.
Gov: Wahh wah wahh wah wah!
Requestor: But the NSA grabs everything and promises it will only look at relevant stuff. We’re just doing the same. We promise we’ll only look at Lerner’s emails. Trust us.
Gov: Wah wahh wah wahh wahh wah!
Requestor: But our promise is as good as the NSA’s. You accept promises from the NSA. Look, we didn’t even have our fingers crossed. You can trust us.
The justice department, where those that enforce the laws declare themselves exempt from those same laws. Go go American banana Republic
Missing the Point...
I’ve seen the similar nightmare with many customers, even those with relatively smaller email databases. I know this nightmare because I’ve contemplated it with many customers in mind.
MS-Exchange, if you use Arcserve backup or Symantec – you have two methods of backup – whole database or item by item. If you’re really, really cheap, you use Windows backup to a flat file, and then backup that flat file backup of the entire database.
If your boss is really cheap and the procurement process is a nightmare, you probably also run Exchange 2003 with some limitations on maximum database size.
“Prior to SP2, each database was limited to 16 GB of total physical size. SP2 increases the licensed database size limit for Exchange Server 2003 Standard Edition from 16 GB to 75 GB; the default configured database size limit will be 18 GB”
So… now the boss won’t spring for fancy backup software or bigger servers, or upgrade the Exchange software or hardware – which by the way is a project, and has to be planned and budgeted like most projects including man-hours.
So you have a backup that’s permanently archived every year-end (not too cheap) but to recover that year’s emails (assuming 1.5GB is a year or more of email, so January each year has not slipped off the abck end).
You will have to:
Create a domain controller first – create one of those on the network, then take it offline and put it on this private network.
Create a new Exchange server as part of this private network, join the domain.
Load the backup tape onto this private domain – oh wait, now the backup physical server with tape drive has to be transferred to the private network temporarily.
You’ve restores the email database, now a client on the private network, running Outlook, has to connect to the Exchange server and copy the emails down to a PST.
Repeat for each year relevant to the investigation.
What do you mean you don’t have Exchange 2003 install media any more, or SP2? it was available for download when Joe set up the servers in 2004.
What do you mean you need 3 free machines to do this, plus a SCSI card for the tape drive, or you have to take our backup process offline?
This assumes to the backup software has not changed since the tape was made or the current product can read the old stuff. If they switched to VMware and VeeAm, did they think to keep the old tape hardware? (There’s plies of reel tapes and punch cards out there in archiveland which are equally useful).
Whenever I dealt with clients with this sort of issue, the general rule was – your backup of Exchange is ONLY useful if the building burns down and you need to pick up where you left off. Anything else will be incredibly painful, expensive, and slow.
Re: Missing the Point...
This is why you don’t use proprietary technology: I have Unix backups from thirty years ago that I can read just as easily today as I could then. (They’ve been transported from 9-track tape to 8mm tape to USB drive, but the bytes are as-written all those years ago.)
I can also USE everything in them because everything in them is in open formats.
Once you marry yourself to proprietary formats, protocol, software, hardware, you’re guaranteeing yourself future headaches. So don’t do that. Ever.
Sounds like aiding and abetting to me. Don’t lie down with these dogs and you won’t get up with fleas.
I totally don't believe in conspiracies, but here's proof of a conspiracy...
How many articles is Timmy going to start by disavowing his generalized misconception of what “conspiracy theories” are before the functioning literates at Techdirt realize how absurd it makes them come across? Can I filter out articles by him? Please?
Re: I totally don't believe in conspiracies, but here's proof of a conspiracy...
Can I filter out articles by him? Please?
Sure. There’s this text that start with “by”. If you see “Timothy Geigner” next to that, then don’t read the story.
Re: Re: I totally don't believe in conspiracies, but here's proof of a conspiracy...
I was thinking more of a kiddies table area, where people can spend all day talking about how totally wrong Alex Jones was about black helicopters. Scrolling past Tim’s articles do Techdirt a huge disservice imo.
Re: Re: Re: I totally don't believe in conspiracies, but here's proof of a conspiracy...
Now I’m confused. Are you an Alex Jones loving crazy person, or someone who thinks ANY claim of conspiracy is crazy?
Either way, you’re wrong, so feel free to go away….
Re: Re: Re:2 I totally don't believe in conspiracies, but here's proof of a conspiracy...
My problem is, as stated before, Tim’s generalized misconception of what “conspiracy theories” are and his constant denial of being a “conspiracy theorist.” Which has nothing to do with the 2 viewpoints you ascribed to me, and your conclusion that I’m “wrong.” Do you often tell people who confuse you to “go away?” How does someone such as yourself learn anything?
I know at least one fruit based tech company that keeps its backups ON THE SAME HARDWARE as the live data.
And that is secured by an unbelievably feeble XML front-door that anyone could access..POOF! all your iSongs are gone forever…..
The backup systems are only designed for use in situations where they didn’t lose the original data on purpose.
I’d been wondering if this wasn’t the case at times.
“It’s too hard” isn’t an excuse; this is (however silly the directions it goes sometimes) a Congressional investigation – *make* it easier.
Or do it anyway, chalk it up as a learning experience, and use a better backup system.
Didn’t the DOJ also say it would be too hard to backup metadata evidence in the Jewel vs NSA unconstitutional spying case? All this evidence destruction isl rather convenient for the Executive Branch.
I’m pretty sure destruction of evidence is a crime. Possibly a felony charge. Our government is staffed by felons.
There was no server crash. They just use backup tapes that get reused far too often and really small exchange email boxes which means they had to store email on their local systems.