Carbonite - Workarounds to back up executables, videos and other files

Carbonite is the online backup service I've ultimately decided to use to protect my data off-site against those proverbial theft/fire/flood events. Unfortunately a Windows Home Server box, as awesome as it is, just isn't off-site (so it gets destroyed or stolen with everything else).

For me, Carbonite does a great job out of the box - documents, photos, music, program/web application source files - ASCX/ASPX/CS/CONFIG/C/CC/CPP/H/HTML/CSS/JS/PHP/Python (ironically including the .pyc files which are actually almost useless to me) /Ruby/Scala/ASM/eqn/JED, program data files - SQL/YAML/XML/XSD/in/out, text files, PDFs, Mercurial repositories (.d and .i for example), compressed and encrypted files - Zip/7z/TrueCrypt/Axcrypt and the vast majority of the other important stuff.


Of course, it's detailed by Carbonite that they do not automatically back up executable files, video files, or many other kinds of files.

In effect, you manually have to go into the relevant folder, right click and select backup (thankfully Ctrl+A to select all and then backing up does work for all files, just not folders).


Why it's a problem
The core is simply - your complete backup is no longer automatic. That means human error begins to creep in.

I'm a computer science student. I have dozens of different projects and assignments completed over the last 4 years, and I know if I come back to them in 5 years, it will be a lot easier to have their executable forms lying around so I can remember the application context more easily.

These are primarily .exe, .o, .jar, .dev and other miscellaneous file formats. Now I will be clear - Carbonite backs up the source forms as detailed above. I'm just saying I want the whole package because some things like compilers and IDEs can become misplaced, hard to find, etc over time. Stuff disappears from the internet all the time, for example if you missed it one of the major original content portals on the internet - Geocities - closed recently.


Workarounds
So, I said Carbonite does by default back up all zip files. This suggests a relatively simple workaround, just put each important file inside a .zip file. Easier said than done?

We'll need some programmatic way of manipulating these files (otherwise it's literally back to the Windows GUI). 7-zip provides such a useful method.

http://dotnetperls.com/7-zip-examples

7za.exe is the command-line version, this one worked for me (though YMMV)
http://downloads.sourceforge.net/sevenzip/7za452.zip?use_mirror=puzzle

Like many programmers, I see myself as pragmatic (e.g. Rasmus Lerdorf, father of PHP - http://itc.conversationsnetwork.org/shows/detail3298.html so I'm going to just do something simple and easy for me.

My backup strategy for these files is basically to call something like the following on the Windows command line, which will create one big .zip file:

C:\Users\Peter>7za.exe a -r -tzip myfiles.zip *.exe

Since this is fundamentally a 3rd tier backup for me, I'm satisfied even though it will take me a little longer to recover the data, and it's possible I've missed something (or Carbonite changes their program's rules, but I reckon they aren't looking to start a war because they'll only lose customers).

Now again being lazy (though I should say this is in the spirit of automation, removing human error), I'd rather not type that into the command line every time (and if I forget something?), so let's turn this into a file called backup_via_carbonite.bat :

GOTO EndComment
    This BAT-file zips up executable files,
    web site favicons, development files, DLLS,
    installers, compilers and other miscellaneous
    files so they are backed up by Carbonite.
   
    Please run it through YOUR OWN TESTING if
    you plan to use it as part of your backup
    strategy!
   
    Notes: - * is a wildcard meaning match all
           - "a" means create archive
           - the -r recurses through the entire
             folder structure
           - myfiles.zip is the name of the 
             resulting .zip archive
           - the -x!Downloads\* excludes files in
             the Downloads folder, same for AppData
    
    Written by Peter Schmidt
    03/DEC/2010
:EndComment
7za.exe a -r -tzip myfiles.zip *.a *.bak *.cab *.com *.dev *.dll *.exe *.ico *.ini *.jar *.lib *.msi *.o *.win -x!Downloads\* -x!AppData\*

Now I can just double click backup_via_carbonite.bat from Windows, put it in my Startup directory, or in the Event Scheduler so it happens as close to automatically as I'd like it to.

My understanding of .zip is it is not a solid compression format and so should be more resilient to small amounts of data corruption, i.e. one flipped bit will not corrupt on average half the files, just one of them.

Now I did say video files. That's another set of extensions - easy to add but hard to discover. Here's a start:

GOTO EndComment
                This bat-file zips up some video files
                so they are backed up by Carbonite

                Written by Peter Schmidt
                03/DEC/2010
:EndComment
7za.exe a -r -tzip myvideos.zip *.flv *.mpeg *.mpg *.mp4 *.m4v *.qt *.wmv

Of course anyone with a sizable video library will know this won't scale, will consume a ridiculous amount of space and well...just be unwieldy and bad. It's possible on many connections that this file would never even be completely uploaded (or may change too frequently) - resulting in no backup at all!

I don't have the solution, except to say Windows 7 does provide the lovely feature of libraries. Try compositing all your videos into one library so you can use the Ctrl+A backup above. If you store your videos in separate folders or across multiple drives - you'll need to add each drive/folder to the library unless you've got a more creative workaround - good luck thinking =)


Final note: Ironically because Carbonite stores older versions of files (including the .zip file that will now be being regenerated automatically on schedule or when I restart Windows), this solution will end up costing them significantly more storage space, bandwidth and time than if they just gave me, the informed paying customer, the option to back up what I wanted conveniently.

Eventually I might get around to splitting this up into separate archives for separate folders, but I'm probably too pragmatic with too much other stuff to get on with...330MB is not too bad a zip file to upload dozens of times over.

Electronic Voting's Issues

Just another part of my extended memory, someone said we should be going to all electronic voting machines in Australia.

For one very good reason, I believe the majority of electronic voting machines are fundamentally flawed - it is hard to give a reasonable guarantee that:

electronic records are written once and from that point read-only.

This is not an impossible problem, for example, modern database management systems can enforce such permissions (assuming there are no bugs affecting this functionality in the DBMS). Ideally this kind of functionality should be implemented in hardware, but that is generally more expensive.

In short, making such systems secure and reliable is a fundamentally hard problem.


Perhaps the best example of a voting machine done (almost) right, the Sequoia AVC Advantage, is well explained on Security Now! Episode 211 - Hacking Electronic Voting Machines:
Main Page - http://twit.tv/sn211
Transcripts - http://www.grc.com/securitynow.htm#211

The best documentary I've found explaining the problems with Diebold and many other machines is the HBO Hacking Democracy Special:
Google Video - http://video.google.com/videoplay?docid=7926958774822130737#
Main Site - http://www.hackingdemocracy.com/

The Princeton University findings are also good:
http://www.youtube.com/watch?v=aZws98jw67g

And the reason I remembered to write this post, a great post linked by the Electronic Frontier Foundation on Twitter:
http://www.freedom-to-tinker.com/blog/jhalderm/hacking-dc-internet-voting-pilot

Fundamentally in electronic mediums, things like trust are still being developed and are often at best fragile and relatively easily broken and overridden. That's the reason the heart of democracy still in my opinion requires a paper trail, literally a reliable, stable audit trail of every single vote - because paper is still a known medium that is much easier to protect and much harder to tamper with.

jQuery - Freedom?!

Yes it should have been discovered sooner. But now that it has been...jQuery+jQueryUI is sheer brilliance.

I can immediately see this freeing me from the demands of stakeholders - I can just give them the ThemeRoller tool (since like most developers, I tend to have a terrible sense of UI design) and tell them to choose how they want the site to look and get them to send me the URL or files back. (Of course the hard part has always been getting the site to work and keeping it alive and secured from DDoS/XSS/CSRF and other attacks, but now I don't have to worry about things like the style of datepickers!)

So why is it so good? Simply put - an excellent pluggable framework for Javascript that unlocks the power of JS, gives the developer great flexibility, and provides better cross-browser support than I probably ever could!


An example related to my current work, building an ASP.NET MVC2 site on .NET 3.5 (c'mon people, .NET 4.0 has been out for a while now).

All I need to do is include in the Site.Master file something like:

<link href="<%= Url.Content("~/Content/jquery-ui-1.8.5.custom.css")%>" rel="stylesheet" type="text/css" />

<script type="text/javascript" src="<%= Url.Content("~/Scripts/jquery-1.4.2.min.js") %>"></script>

<script type="text/javascript" src="<%= Url.Content("~/Scripts/jquery-ui-1.8.5.custom.min.js")%>"></script>


obviously the actual URLs may be different depending on your project setup, the above come out to something like ../../Scripts/jquery-1.4.2.min.js, but that depends on the context of the request.

Whenever the stakeholders decide to redesign part of the site - it's update 3 files/references and voila! Whenever performance and security fixes come down the pipeline, same quick update! And in web development, it's all about taking DRY to the limit =)


So for an actual example, let's try making a datepicker:

1. Add the javascript for the datepicker itself, for a basic Aussie date format:
   
<script type="text/javascript">
  $(document).ready(function () {
      $(".datepicker").datepicker({
          dateFormat: 'dd/mm/yy',
    });
});
</script>

   
2. Add the "datepicker" class to the relevant div or input type="text" area
   
<input type="text" id="releaseDate" class="datepicker"/>

   

Of course for full details of what's possible, see the jQuery DatePicker page. http://docs.jquery.com/UI/Datepicker

But remember I said ASP.NET MVC2 ?
That's something like (simplified):

<div class="editor-field">
       <%: Html.TextBoxFor(model => model.Title) %>
</div>

i.e. using auto-generated view code (literally right-click on a controller ActionResult and click "Add View") which uses lambda expressions to reference model attributes, or something like that.

The key point is MVC2 abstracts the HTML using the built-in (and extensible) HTML helper classes. Seems like it's time to write one of those to take advantage of jQuery!

Update 5:29PM - An excellent post by Steve Michelotti covers this very well:
http://geekswithblogs.net/michelotti/archive/2010/02/05/mvc-2-editor-template-with-datetime.aspx

Google Wave killed to focus on the Aussie NBN?

Opinion

It is well known that Google recently killed their implementation of Wave. It is also well known that Australia’s current Labor government has currently promised to build a $43 billion Fibre-To-The-Home National Broadband Network (NBN). The first nodes are now operational in Tasmania, with (hopefully) much more to come.

It has recently been announced that the NBN will improve its coverage to 93% of the population, and deliver even higher speeds of up to 1000 Mbps (up from 100 Mbps). That’s the same speed promised by Google’s experimental network in the USA for 50,000 to 500,000 people, except it’s potentially on the scale of up to 93% of 22.4 million people. I don't think anyone at Google Australia missed this...the right kind of sufficiently pure fibre should be capable of 50Gbps or maybe even much more.

Even if only a small portion of Australians actually take up the offer it could easily be on par with the US experiment. If Google, with some of the cleverest engineers on the planet figures out killer apps for Aussie blokes and sheilas and their real families...there's an easy roadmap for taking the experiment to a level that's an order of magnitude larger.

However there’s a huge risk – an upcoming election on August 21^st, 2010. If the Coalition wins the election, it has promised to kill the NBN, replacing it with a $6.3 billion patchwork of technologies. What does this mean for corporations like Google? Uncertainty - perhaps the single most destructive force for business. Why? Because as a business you can't easily justify the longer term investments that are often the most productive and profitable. For Google that might mean committing engineers to bandwidth-intensive next generation consumer and enterprise apps instead of whatever more mundane tasks need doing (like babysitting Wave?).

What would a Labor victory deliver? Governments tend to move slowly, by the time of the next election enough of the network will actually have been built (since we're already about 1 year into the estimated 8 year rollout period) that it will be effectively completed, almost no matter what happens in the election after.

So was the death of Google Wave a coincidence or a masterful strategic move?

Killing Wave frees up dozens, perhaps hundreds of Google employees in Sydney who were working on Wave, likely as their 80% project, to think about and work on architecting, designing and constructing the next generation of applications for the post-NBN world. And that means these Googlers and anyone else in the technology industry may have a huge experimental playground for building things like GoogleTV, videoconferencing, enterprise and cloud services, maybe even healthcare and educational applications or entirely novel things we can barely imagine.

Of course, it depends on how Australians vote on August 21st. Who knows what some of the best engineers and PhD’s could come up with on the next generation of network?

Smart people + passion + bandwidth = ?

Personally I’m excited to think of what might be in the pipeline. Whether this was a happy accident, a random coincidence or a master stroke of genius, I say bring on the innovation!

Why the Coalition's $6.3bn broadband plan is a waste of money

According to The Australian, under a Coalition broadband policy, we'll be getting:

1. "$6.31 billion over seven years"
2. "$2.75bn to the construction of an optic fibre backhaul network but also relies on at least another $750 million from the private sector"
3. "$1bn in grant funding for a rural and regional wireless network"
4. "$1bn to build a wireless network in metropolitan Australia"
5. "Satellite coverage for the remaining 3 per cent of the population will receive a $700m boost"

http://www.theaustralian.com.au/national-affairs/abbotts-broadband-lite-plan-no-game-changer/story-fn59niix-1225903503971

Why is this a waste?

Firstly the Coalition could have had another $6.3bn to spend on other things like roads, rail or healthcare.

Secondly they're building a "optic fibre backhaul network" - that sounds pretty similar to what MFS-WorldCom, New Edge Networks and others in the United States spent over $90 billion on in their technology boom, depending on who you believe much of that is still wasted "dark fibre".

Thirdly while I'll agree a wireless network makes sense in rural areas with very low populations, it doesn't whenever you get more than a few simultaneous users. On a properly built fiber network, you can get the full speed - Labor and the NBNCo are promising 100 Mbps minimum if you are willing to pay for it. That is a guarantee.

On a wireless network (and the Telstra/Optus HFC "Cable" networks) you have what is called contention - when other users use the same network the performance you get degrades. It's not uncommon for many users to degrade the performance of WiFi 802.11b networks from 11Mbps to 1Mbps or even lower making the network unusable (try this in a university library for example) even with the best algorithms to share the bandwidth.

And consider for a second - everyone on a wireless network must obey the rules - anyone with a misbehaving piece of equipment (or even someone who was deliberately being malicious) could interrupt service - not for one user but for dozens or hundreds - at once.

That makes applications like healthcare monitors for elderly citizens impossible - it's simply too risky. It also makes latency sensitive applications, like videoconferencing or online gaming, much less engaging (if you've had dropped telephone calls in the past - how often have you dialled the person back and how did you feel?)

Fourthly, even if you were *extremely* optimistic about the Coalition's wireless technology, it might be capable of a peak theoretical 1Gbps with the commercialisation of the still in progress 802.16m WiMAX. Now split that between 100 users per deployed node (10Mbps?). Many metropolitan and regional users today have ADSL2+ DSLAMs featuring up to 24Mbps. However you cut it, the proposed wireless technology either doesn't exist today or is inferior to existing wired technology, and vastly inferior to a fiber network.

Finally satellite coverage is receiving a $700 million boost. The NBNCo is planning to deploy 2 satellites for rural Australia at $500 million each, providing better bandwidth and redundancy in the event of one failing. There aren't that many makers of satellites, it sounds like the Coalition might end up wasting $200 million right off the bat and get just a single satellite.

Now I should be clear, I believe in the private sector moving forward with wireless deployment, it has great applications, especially in the mobile phone/laptop/netbook/tablet space. But wireless spectrum is still relatively scarce and relatively limited - and for the forseeable future it will remain so. There's are many more good reasons the IT industry has wholeheartedly slammed this Coalition broadband plan.

Twitter!

So if you stumble across this place and decide for some reason you'd like to contact me, the simplest way is via @pzrq on twitter =)

P.S. It never ceases to amaze me the treasure trove Wikipedia has become (most of the time) :

http://en.wikipedia.org/wiki/Blog

http://en.wikipedia.org/wiki/Max_flow_min_cut_theorem

Homeschool education

To homeschool or not to homeschool? What generates the better outcome for your children?

TLDR version: We just don't know, but here's my thoughts on how you might begin, if it's the right question to ask in the first place.

Why should parents have a choice? Because parents who are passionate and motivated will assess the options available to them, the current life circumstances such as career, income, educational attainment and marital status, and only then be able to make an informed decision about what they believe is best for their children and themselves collectively (that's why it's called a family).

This started with frankly a little offhand tweet of mine:

The biggest problem w #homeschool is your kids aren't gonna get the practical social knowhow they need to be part of any real team/workplace

And a reply from @farmmom4him:

@pzrq My Hsed children R able 2 interact in conversation w/ all ages, not jst their same aged peers. cnt say that 4 PS children. #homeschool

Which to my mind prompts the question I frankly was ill-equipped to even try to fully articulate, let alone answer before bed and with work in the morning...

What effect does homeschooling have on a child's social outcomes? And how does that effect compare to that of public/private schools?

What do I mean by social outcomes? I think I've defined it but I'll flesh it out as effective integration into teams and workplaces, whether built on surfing, tennis, soccer, climbing a mountain or other sporty things; debating or writing, software or hardware development, diplomatic or trade negotiations, banking and stockbroking, politics, advertising, podcasting, or whatever other collaborative endeavours are needed in the modern world...

...through effective social interaction.

Unfortunately that is not a simple question...it sounds like a topic of vague abstract notional institutional thinking, or perhaps a long term research thesis (and if I ever do one of those, it'll almost certainly be on something in technology). We also have to immediately discount all the anecdotes and stories, things like "cnt say that 4 PS children" - because I can with overwhelming probability guarantee there are some who can!

To be clear, social outcomes are something we can't easily measure. It's not like financial outcomes or standardised test scores which are both very well defined - it's easy to see who does better because they get a better result in the form of earning more money or a better mark (e.g. for a good overview see this HSLDA report by homeschooling advocate Brian D. Ray)

Now why did I start defining my question with "what effect does"? It is asking not the relatively easy question of correlation, but the much more difficult one of causation! To those unfamiliar...here's a clever example from Steve Gibson on Security Now! Ep 209:

imagine [you're]...watching the street...in New York, and [you] noticed that suddenly everyone put their umbrellas up and, oh, look, then windshield wipers all began going on the cars. Well, if you didn't know any better, you didn't understand anything about what was really going on, you could say that raising umbrellas caused windshield wipers to go on.

Causation...

Because it's not too hard to believe society has defined the default education as a public (or private) education by teachers in classrooms and lecturers in theatres, which means only people with a motive will seriously commit to an alternative such as homeschooling. People with a motive are probably more passionate and more concerned for their children's education, so much so that they wish to spend a significant part of their lives participating in precisely that - being a teacher/guide/mentor to their children.

Unfortunately, none of the mentioned homeschool material I've come across, for example:

• HSLDA Homeschool Progress Report, 2009 http://www.hslda.org/docs/study/ray2009/2009_Ray_StudyFINAL.pdf
• NCES Homeschooling Brief, Dec 2008 http://nces.ed.gov/pubs2009/2009030.pdf
• NCES Homeschooling Report, from 2003 data http://nces.ed.gov/pubs2006/homeschool/
• Home-based education effectiveness research and some of its implications, Dr Doland Meighan http://www.education-otherwise.org/HE/Home-based%20education%20effectiveness.pdf
• Google Scholar, e.g. http://scholar.google.com.au/scholar?q=homeschooling&hl=en&btnG=Search&as_sdt=2001&as_sdtp=on

as far as I can tell addresses this much more difficult causal question, but I'll keep my eyes out.

I think for the time being it's consigned to the heap of often extremely subjective questions like

What is the best diet?

I'm also skirting the other raised issue of how do you measure social outcomes...anyone should be able to come up with a counter example to simple things like the number of friends you have, the number of extracurricular activities you participate in, or how satisfied you personally feel about your relationships with other people.

I think the real reason questions like mine have not been seriously asked, let alone answered is simply that homeschooling simply involved too small a number of people for as long as I've known, and the statistics appear to show a trending increase in recent years. I'm sure historically there were much more interesting and accessible questions available for anyone passionate about education.

I'll close with what I think is a remarkable insight - perhaps this is the wrong question to be asking anyway (if you have to ask it) :

The answer will not be heartening to obsessive parents : in this case, school choice barely mattered at all...a student who opted out of his neighbourhood school was more likely to graduate whether or not he actually won the opportunity to go to a new school...the students - and parents - who choose to opt out tend to be smarter and more academically motivated to begin with. But statistically they gained no benefit by changing schools.

-Steven D. Levitt, Freakonomics, International Edition, HarperCollinsPublishers, Ch5 What Makes a Perfect Parent?

I'll forgive anyone for thinking much ado about nothing, but I believe that's how real knowledge begins. You must ask the right questions.