Search This Blog

Sunday, October 30, 2011

How to migrate mail from one Gmail account to another on Windows, including Chats


This is the Microsoft Windows version of this article (though it contains screen shots from Mac). The Mac OSX version can be found here.

Recently at work, we got a new Gmail domain and had to switch from one Gmail account to another.

You can migrate your mail easily using POP3, but POP3 won't get your chats or preserve your labels. The superior solution is IMAP, even if it's a little more cumbersome. And even if you have already migrated using POP3, IMAP will still allow you to retrieve your chats and work for you on a tag-by-tag basis.

IMAP requires bit of a work-around—you must first fully download all your files to your machine, and only then can you migrate them back again into your new Gmail account.

Here's everything you need to get set up.

First, the downloads.
  1. Download Mozilla Thunderbird and install on your machine.
  2. Download and unzip the toIMAP files (we'll set these up later)
Now log in to your usual (old) Gmail account.

Then:

1. Click the gear icon in the upper-right corner and select Mail settings:


2. Click the Forwarding and POP/IMAP tab, scroll down to IMAP Access, and make sure it is turned on:


3. Click Labels and make sure that Chats and any other labels you want migrated are checked. (Note: If you already used the POP3 solution, you might want to limit only to Chats.)


4. Open Thunderbird, click Create a new account and use the address you want to pull the mail from, then click Continue:


Note: The wizard should smartly detect the proper settings; if not, you can click Manual config and enter the IMAP and SMTP settings shown below:


5. Click Create Account and let Thunderbird start downloading; this may take hours, and it goes without saying that the more mail you have, the longer the process will be. If you have to shut down for a while, to easily restart where you left off, just open Thunderbird, right-click, and click Get Messages in the context menu:


6. Log in to your new account and repeat steps 1 and 2 above for it. Make sure that you re-create any custom labels you used in your old account:


Now... go find something else to do for a few hours and come back to step 7 after Thunderbird (eventually) finishes.

7. Yay! Your e-mail finally downloaded! Rejoicing, right-click your account name, and click Settings (last option in the screen shot for step 5). Navigate to Server Settings, and copy the Local directory path at the bottom.


8. Open a new folder, and paste this address in the location bar. Keep this window open, as you'll need it for the following steps:


9. Go to Start > All Programs > Accessories > Notepad and start the program. Use it to open the toIMAP.cfg file in the toIMAP folder you downloaded to your Desktop. Make sure your configuration file is set up like the one below, subbing in your username and password:
{
     'host' : 'imap.googlemail.com',
     'user':'YOU@gmail.com',
     'password':'YOURPASSWORD',
     'ssl': True
}
10. Save the file and close it. Your configuration file is now set; all that is left is to run the file.

Go to your Windows Start menu and type cmd.exe in the search box, pressing ENTER. The Windows command-line prompt will open.

You'll need to be at the same level as the file; for Microsoft Vista, the path would be (where "YOU" is your username): C:\Users\YOU\Desktop\toIMAP. To do this, you'll need to change directories (cd) into the folder on your desktop:
cd C:\Users\YOU\Desktop\toIMAP
To run the uploader script, you'll use the following convention while still sitting at the level in step 9:
toIMAP.exe -m path_i_just_copied/FOLDER_NAME -f FOLDER_NAME
For example, if I want to upload the folder "admin stuff" (shown in step 8), I would type (surrounding my file path with quotation marks to bypass the space between "admin" and "stuff"):
toIMAP.exe -m "C:\Users\YOU\AppData\Roaming\Thunderbird\Profiles\76sexgps.default\ImapMail\imap.googlemail.com\admin stuff" -f "admin stuff"
Some Gmail flags like Starred may need to omit the -f "admin stuff" part.

To specify a different Gmail tag than the name of the Thunderbird folder, just pass -f the new name, making sure that the tag already exists in your new account per step 6.

For example, to move your chats, you will not be able to use the same "Chats" name for your tag as that is reserved by Google. You'll need to create a new tag in your new account—I used "Archived Chats"—then import using the additional -f flag:
toIMAP.exe -m "C:\Users\YOU\AppData\Roaming\Thunderbird\Profiles\76sexgps.default\ImapMail\imap.googlemail.com\[Gmail].sbd\Chats" -f "Archived Chats"
If your upload gets interrupted at any point, see the toIMAP directions on how to resume where you left off.

11. Repeat step 10 for as many folders as you need to migrate. Once you have gone through every folder you want backed up, you're done. Bliss is yours.

How to migrate mail from one Gmail account to another on Mac OSX, including Chats


This is the Mac OSX version of this article. The Windows version can be found here.

Recently at work, we got a new Gmail domain and had to switch from one Gmail account to another.

You can migrate your mail easily using POP3, but POP3 won't get your chats or preserve your labels. The superior solution is IMAP, even if it's a little more cumbersome. And even if you have already migrated using POP3, IMAP will still allow you to retrieve your chats and work for you on a tag-by-tag basis.

IMAP requires bit of a work-around—you must first fully download all your files to your machine, and only then can you migrate them back again into your new Gmail account.

Here's everything you need to get set up.

First, the downloads.
  1. Download the Mac edition of Mozilla Thunderbird and install on your machine.
  2. Download and untar the toIMAP files (we'll set these up later)
Now log in to your usual (old) Gmail account.

Then:

1. Click the gear icon in the upper-right corner and select Mail settings:


2. Click the Forwarding and POP/IMAP tab, scroll down to IMAP Access, and make sure it is turned on:


3. Click Labels and make sure that Chats and any other labels you want migrated are checked. (Note: If you already used the POP3 solution, you might want to limit only to Chats.)


4. Open Thunderbird, click Create a new account and use the address you want to pull the mail from, then click Continue:


Note: The wizard should smartly detect the proper settings; if not, you can click Manual config and enter the IMAP and SMTP settings shown below:


5. Click Create Account and let Thunderbird start downloading; this may take hours, and it goes without saying that the more mail you have, the longer the process will be. If you have to shut down for a while, to easily restart where you left off, just open Thunderbird, right-click, and click Get Messages in the context menu:


6. Log in to your new account and repeat steps 1 and 2 above for it. Make sure that you re-create any custom labels you used in your old account:


Now... go find something else to do for a few hours and come back to step 7 after Thunderbird (eventually) finishes.

7. Yay! Your e-mail finally downloaded! Rejoicing, right-click your account name, and click Settings (last option shown in the screen shot for step 5). Navigate to Server Settings, and copy the Local directory path at the bottom.


8. Open Finder, and select Go > Go To Folder, then paste the path you just copied. Keep this window open, as you'll need it for the following steps:


9. Open a new terminal window and type the following lines on your Mac:
cd ~/Desktop/toIMAP
vim toIMAP.cfg
i
The "i" puts you in insert mode; make sure your configuration file is set up like the one below, subbing in your username and password:
{
     'host' : 'imap.googlemail.com',
     'user':'YOU@gmail.com',
     'password':'YOURPASSWORD',
     'ssl': True
}
Then press ESC to exit insert mode, and type:
:wq!
to write and quit (save and exit) your configuration file.

10. Your configuration file is set; now all that is left is to run the file. Unfortunately, it does not appear that the script is smart enough to parse all folders at once, so you'll need to parse a single folder at a time, hence the opened Finder window in step 8 so you can reference each of your mail folders.

To run the file, you'll use the following convention while still sitting at the level in step 9:
python toIMAP.py -m path_i_just_copied/FOLDER_NAME -f FOLDER_NAME
For example, if I want to upload the folder "admin stuff" (shown in step 8), I would type (using a backslash to escape the space between "admin" and "stuff"):
python toIMAP.py -m ~/Library/Thunderbird/Profiles/76sexgps.default/ImapMail/imap.googlemail.com/admin\ stuff -f "admin stuff"
Some Gmail flags like Starred may need to omit the -f "admin stuff" part.

To specify a different Gmail tag than the name of the Thunderbird folder, just pass -f the new name, making sure that the tag already exists in your new account per step 5.

For example, to move your chats, you will not be able to use the same "Chats" name for your tag as that is reserved by Google. You'll need to create a new tag in your new account—I used "Archived Chats"—then import using the additional -f flag:
python toIMAP.py -m ~/Library/Thunderbird/Profiles/76sexgps.default/ImapMail/imap.googlemail.com/[Gmail].sbd/Chats -f "Archived Chats"
If your upload gets interrupted at any point, see the toIMAP directions on how to resume where you left off.

11. Repeat step 10 for as many folders as you need to migrate. Once you have gone through every folder you want backed up, you're done. Bliss is yours.

Saturday, October 08, 2011

Javascript: A Better typeof -- Accurately Determine the Type of Variable (Array, String, Boolean, Number, Function, etc.)

If you have ever needed to determine the type of a given Javascript variable, you have probably found the native typeof somewhat inadequate.

Here's a complete solution, adapted from one originally seen on JavaScript Garden:
var realTypeOf = function(obj) {
    return Object.prototype.toString.call(obj).slice(8, -1);
};

// usage
realTypeOf('hi there'); // String
realTypeOf(true); // Boolean
realTypeOf({}); // Object
realTypeOf([]); // Array
realTypeOf(function() {}); // Function
realTypeOf(new Date()); // Date
realTypeOf(/[0-9]/); // RegExp
realTypeOf($('does-not-exist')); // Null
realTypeOf(document.body); // HTMLBodyElement
realTypeOf($('output-box')); // HTMLDivElement
realTypeOf($$('a')[0]); // HTMLAnchorElement
realTypeOf($$('link')[0]); // HTMLLinkElement
realTypeOf($$('script')[0]); // HTMLScriptElement
realTypeOf(999); // Number
realTypeOf(Math.min(3, 7)); // Number
Try it out in real-time with this JSFiddle example set.

Sunday, October 02, 2011

PHP: Simple and Effective Way to Validate Email Addresses


There are a number of regular expression solutions that have been written to test the validity of user-entered email addresses in PHP, some of them quite good. However, the most consistent and reliable way to validate email addresses takes advantage of PHP's built-in functionality.

filter_var does just what it says: it filters a variable using a variety of pre-defined filters (see the complete list of filters categorized by type); when you are validating email addresses, the filter you want is FILTER_VALIDATE_EMAIL:
if (!filter_var($_POST['email'], FILTER_VALIDATE_EMAIL)) {
     exit('E-mail is not valid.');
}

Saturday, August 27, 2011

Object doesn't support property or method 'createContextualFragment': Solved Internet Exporer 9 JavaScript Error


Object doesn't support property or method 'createContextualFragment'

I use a rich-text editor on my site, and received this JavaScript error when trying to view it on IE9. An easy solution exists, however, which I found in a discussion thread; simply add the following lines of code at the top of your file or before the method where you use createContextualFragment which makes it available to the Range object via prototyping.

if ((typeof Range !== 'undefined') && !Range.prototype.createContextualFragment) {
    Range.prototype.createContextualFragment = function(html) {
        var frag = document.createDocumentFragment(); 
        var div = document.createElement('div');
        frag.appendChild(div);
        div.outerHTML = html;
        return frag;
    };
}

Saturday, August 13, 2011

ChromePHP: Google Chrome's PHP Debugging "FirePHP" Extension


Firebug is by far the most popular developer's tool for Firefox. FirePHP is another well-known add-on that allows for debugging PHP. Unquestionably great dev tools.

Google Chrome's built-in debugger is arguably parallel in features and performance to Firefox's Firebug. A lot of devs like using Chrome, but they don't know that they can also debug their PHP in it as well. An extension called ChromePHP bridges that gap.

Still haven't heard of any browser console debuggers for Python in Chrome. Looks like your only choice of a command-line debugger for now is Firefox/Firebug using FirePy (or git clone here), though it probably wouldn't take too much work to use Python's built-in logger to capture and log to the console via JavaScript.

Wednesday, July 06, 2011

JavaScript Basics (HTML, CSS, Selectors, Events)

Note: This entry was first published on the company wiki at IWS to better help Python devs learn Javascript.

It has been here slightly modified to omit or work around IWS-specific customizations, with source code supplied for fromJSON(), toJSON(), and toObject().

You may also want to download MooTools to try some of the examples here, or follow along in your own JSFiddle (where MooTools is the included default).


HTML is to JavaScript what the Database is to Python

Just as a guide on the essentials of using Python for a web-based application would be incomplete without at the very least mentioning how it interacts with the database, so too a guide on JavaScript is incomplete without understanding how it interacts with HTML.

To understand JavaScript well, one should have at least a basic understanding of both HTML and CSS, as any web application is made of web pages, and web pages are written in HTML which is itself styled with CSS.

Understanding the power of interactive JavaScript also involves a basic understanding of events: when the user clicks a mouse or enters commands on the keyboard or drags a finger across a screen or track pad, the interface magically responds. The magic between the user action and the result is called an event, and the way JavaScript understands events is by “listening” to an HTML element: by attaching an “event listener.”

We will discuss all these things, and more, in greater detail.

Brief Glance at HTML: Elements Nested Inside Elements Nested Inside...

Let’s begin with a very brief look at HTML, short for “hyper-text markup language.” A typical web page has a doctype declaration that explicitly tells the browser how to render the page. Following this declaration, the entire document is surrounded by opening and closing html tags.

Nested inside these outer tags are two other inner sets of tags: an opening and closing head tag followed by an opening and closing body tag. Inside the head tag, things such as the title of page (displayed in the browser history and on the browser tab), any meta data, links to any style sheets, and links to any JavaScript files are specified. The body section that follows contains the actual elements that make up the visible part of the web page.

Here’s a skeleton of a web page using the HTML5 doctype:
<!DOCTYPE html>
<html>
    <head>
        <title>Sample Page</title>
        <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
        <link rel="stylesheet" type="text/css" href="css/some-css-file.css" />
        <script type="text/javascript" src="js/some-js-file.js"></script>
    </head>
    <body>
        <p>
            I am a single-sentence paragraph on the page.
        </p>
        <p>
            I am another paragraph on the page. I contain two
            sentences.
        </p>
        <p>
            I am a third paragraph on the page. I contain three
            sentences. See?
        </p>
        <div>
            I am a generic “div” element that contains a
            <a href="http://www.some-link.com">hyperlink</a>,
            otherwise known as an “anchor” tag, hence the
            opening and closing “a” brackets.
        </div>
        <!-- I am a multi-line HTML comment that the browser
        will ignore: I exist only to comment out sections of
        HTML markup or to be read by human “marker-uppers”
        and “code sniffers.” -->
    </body>
</html>
Within the opening and closing head tags in the skeleton page above, we define the title, the mime type, then link to an external style sheet, then an external JavaScript file.

Notice that we link to the style sheet before linking to the JavaScript: it would technically work in any order, but best practices places a priority on the style sheets first so that the visual elements will not be delayed while the JavaScript is loading.

We could technically include several JavaScript files on a page, and the thing to remember is that order can matter. That is, all JavaScript files have access to the JavaScript in all other JavaScript files loaded in the page (and any inline JavaScript on the page), but if a file gets instantiated immediately and relies on code in another file, the file containing the required code has to be loaded first.
<head>
    <script src="js/file-containing-needed-code.js" type="text/javascript"></script>
    <script src="js/file-that-needs-above-code.js" type="text/javascript"></script>
</head>
It’s a matter of resolving dependencies: add the file you need to make a second file work first, then add the second file.

The browser is to JavaScript what the server is to Python: One other point to notice is that the HTML page links to the JavaScript file, which is loaded and interpreted on page load. That explains why, unlike with Python, you have to refresh the HTML page before you will see any changes you have made to the JavaScript file. Whereas the server interprets the Python file, it is the browser that interprets the JavaScript file: different run-time environments!

Returning to the skeleton page, our opening and closing body tags contain three paragraph elements, a div element, an anchor element (nested inside the div), and a comment node.

We now have enough pieces in place to look at CSS, short for “cascading style sheets.” We’ll talk about the “cascading” part in just a moment, but our first order of business is understanding how style sheets work in general.

Cascading Style Sheets (CSS): Selecting by Tag Name

Style sheets consist of key-value pairs, where the key is a predefined CSS property and the value is the setting one wishes to apply for that property. These pairs are assigned on an element-by-element basis.

For example, if we wanted to separate all the paragraph tags by 28 pixels and draw a 1px border around them that was bright orange, we could specify that tag (p) followed by curly brackets that contained the key and value pairs we wanted:
/* Only C-style, multi-line comments are permissible in CSS.
#f91 is short for #ff9911, the hex value for a vivid
shade of orange. */
p {
    margin-bottom: 28px;
    border: solid 1px #f91;
}
The above specification could be saved in a text file named “some-css-file” and given a “css” extension; we could then park it inside the “css” directory, and then the <link> tag in our example page would “pull it into the page” and apply a bottom margin of 28 pixels to all paragraphs drawing an orange box around each one.

CSS: Syntactic Significance of Commas and Spaces

If we wanted, we could specify additional elements as well, separated by commas:
p, div {
    margin-bottom: 28px;
    border: solid 1px #f91;
}
The example above would apply the same style rules to all paragraph and all div elements on the page.

By contrast, if we used a space instead of a comma, we wouldn’t see any visual difference in our example page from unformatted HTML. Do you know why?
p div {
    margin-bottom: 28px;
    border: solid 1px #f91;
}
Whereas the comma forms a list of elements, a space says “for all div elements inside of paragraph tags, apply these rules.”

This particular example doesn’t make much sense, as div tags would rarely be nested inside paragraph tags anyway (though the reverse might well be true), but in any case, we don’t have any divs inside paragraph tags on the page and thus this rule would simply be ignored.

CSS, then, can use element tag names to target an element. But that’s pretty limited, so there are at least two other very common ways to target an element with CSS. Let’s modify our HTML page slightly to include id and class attributes.
<body>
    <p class="paragraph">
         I am a single-sentence paragraph on the page.
    </p>
    <p class="paragraph bold large" id="special-paragraph">
        I am another paragraph on the page. I contain two
        sentences.
    </p>
    <p class="paragraph">
        I am a third paragraph on the page. I contain three
        sentences. See?
    </p>
    <div id="special-div">
        I am a generic “div” element that contains a
        <a href="http://www.some-link.com">hyperlink</a>,
        otherwise known as an “anchor” tag, hence the opening
        and closing “a” tags.
    </div>
</body>

CSS Classes and Ids

Just like variable names, we can name classes and ids anything we want. The only rule is that ids are mutually exclusive—there can only be one element named a given id on a given page—whereas classes can be used over and over, and, as in our second example, there can be multiple classes separated by spaces for a single element.

Accordingly, our example employs three instances of the class “paragraph,” a single instance of the classes “bold” and “large” each, and two unique ids, one named “special-paragraph” and the other “special-div.”

Like variables, we could just as easily have named them “jack-and-jill” or “YellowSunshine” or anything else in the universe we wanted, so long as they (1) started with a letter and (2) were alpha-numeric (including underscores and dashes).

To specify a style rule for a class in CSS, the class name is signified by a dot or period:
.paragraph {
    /* some rules */
}
To target an id, we need to use the pound sign:
#special-div {
    /* some rules */
}
As with our tag name examples above, multiple elements can be specified with a comma, separated by a space to target selected child elements inside a parent element, or mixed and matched in some combination thereof:
#my_div .children, h4, h2.some-class-on-h2-elements, p#a-unique-paragraph-id, .catch-all {
    /* rules to apply to this assortment of elements */
}

Wrapping Up: CSS Tags, Classes, and Ids

There are other selector methods used to target elements (see Mootools: Complex CSS3 Expressions), but a basic understanding of CSS only requires that we know we can use tags (with no preceding punctuation)...
p {
    /* some rules */
}
Classes (with a dot)...
.some-class {
    /* some rules */
}
And ids (with a pound sign)...
#unique-id {
    /* some rules */
}
And that we can target multiple elements at a time with commas or drill down into a parent element using spaces.

To summarize what we’ve covered thus far, style sheets specify properties for one or more elements (using tag names, classes, and ids) by means of CSS properties (keys) and user-specified values.

Getting Ready to Cascade: External, Internal, Inline

Additionally, there are three ways that styles can be declared. We’ve covered the first one already:

(1) In the <head> section of the page, we can use a <link> element to link to an external style sheet.

(2) In the <head> section of the page, we can use internal <style> tags:
<head>
    <style type="text/css">
        div {
            font-size: 14px;
            color: #eedeca;
        }
        .some-class {
            font-weight: bold;
            font-family: verdana;
            background-color: #e2e3e1;
        }
    <style>
</head>
Side note: You can also write internal JavaScript in the <head> or <body> sections using opening <script type="text/javascript"> and closing </script> tags.

(3) Directly on any element in the <body> section using an inline “style” attribute:
<p style="font-style: italic; margin-left: 30px; font-size: 18px; font-family: georgia; color: blue;">
    I am an italicized paragraph with a 30-pixel left margin.
    My font size is now 18 pixels, my font family is Georgia,
    and I’m feeling blue.
</p>

I am an italicized paragraph with a 30-pixel left margin. My font size is now 18 pixels, my font family is Georgia, and I’m feeling blue.

So now we finally come to understand what the “cascading” part of CSS means. A “cascade” is often used to describe water, as in a “cascading waterfall” that “cascades” from top to bottom. Likewise, CSS has a “cascading” order of overwrite importance rule.

If in my external style sheet, I declare that all paragraph tags should have brown font, but then in the internal <style> tags in my head turn around and say that all paragraph tags should have green font, the browser will render my HTML page with green paragraph text.

But if on one (or more) paragraphs I include an inline style that specifies yellow font, the affected paragraphs will be displayed in yellow font (even though in the internal style rules I specified green and in the external style sheet I stipulated brown). Hence “cascading.”

With a basic understanding of HTML and CSS in place, we’re now ready to talk about JavaScript, formally known as ECMAScript.

ECMAScript a.k.a JavaScript

JavaScript is able to grab and manipulate screen elements; it can also style elements by applying CSS rules directly to them. Both of these are inter-related tasks.

The Crooked Path of the DOM

JavaScript interacts with HTML via the Document Object Model, better known as the DOM.

The DOM is not HTML and the DOM is not JavaScript: the DOM is a tree-like API that a browser layout engine provides JavaScript for working with HTML.

And, through no fault of JavaScript or modern browser vendors, the DOM has traditionally been a particularly “clunky” and comparatively inefficient way of working with HTML, but it’s the only way we’ve got (and HTML5 is working hard to take some of the sting out of this legacy API).

Now is a good time to mention as well that JavaScript itself depends on browser implementation: a typical browser is written in C and the individual browser vendor (like Microsoft or Mozilla) is responsible for writing the parser/interpreter that compiles JavaScript on page load into C. They do this by coding to an interface—coding to the standard laid out by ECMA International that says method x, y, and z should exist and behave in such-and-such manner—but the actual implementation is left up to the individual vendor.

That means that JavaScript just got more complicated: with Python, once you get the server set up like you like it, you code once and you’re done. But while server-side JavaScript does exist and is gaining in popularity, client-side JavaScript is utterly at the mercy of the browser vendor.

Though times are thankfully changing, engrave this maxim into your forehead: not all browser vendors are created equal!

The Crooked Path Made Straight: JavaScript Frameworks To The Rescue

The “clunkiness” of the DOM and the inconsistencies between browsers is where JavaScript frameworks or libraries—like JQuery, Dojo, Prototype, or IWS’s choice of MooTools (short for “My Object-Oriented Tools”)—really shine.

These frameworks are written in pure JavaScript and they smooth out a lot of the inconsistencies between browsers, additionally providing a lot of useful tools that make coding to the browser easier: not quite as smooth as coding to the server perhaps, but definitely a lot closer.

Since we use MooTools at IWS, that will be the framework we’ll look at here.

MooTools makes heavy use of CSS selectors (as the tag, dot, and pound notations we covered above are collectively named) as “handles” to grab hold of elements in the DOM, its selector engine Slick arguably neck-to-neck with JQuery’s well-known Sizzle engine in terms of speed and power. But we don’t care about all that. We just need to know how to use the tools we’ve got to get the job done.

My Object-Oriented Tools (a.k.a. MooTools) and CSS Selectors

In native JavaScript, if we want to grab an element with a unique id, we drill down from the top-level of the DOM, otherwise known as the document:
var myElement = document.getElementById('some-element-id-on-the-page');
In the example above, we assign the private myElement variable (private because of the var keyword) to the return of document.getElementById. Notice that the method name begins with getElement (singular) not getElements (plural). That is because a CSS id is always supposed to be unique, and we are interested in grabbing just one element.

MooTools’ $()

MooTools gives us a shortcut, a shortcut that happens to be shared by virtually all the major frameworks: MooTools gives us the dollar sign:
var myElement = $('some-element-id-on-the-page');
$() is expecting an id, so we don’t need to pass it the # selector—we just pass a string that corresponds to an actual id that lives on the page. As with the native document.getElementById method, the MooTools’ $() returns null if an element is not found (equivalent to None in Python).

null notwithstanding, when testing to see if an element is in the DOM, it is generally best/cleanest simply to test for truthy / falsy:
if (myElement) {
    // myElement exists: do something with it 
}

if (!myElement) {
    // it doesn't exist: create it
    myElement = new Element('div');
}
The DOM allows native JavaScript to gather together a group of elements in several ways, such as:
var myInputsNamedEmail = document.getElementsByName('email');
var myParagraphElms = document.getElementsByTagName('p');
These methods all begin with getElements (plural) and, like an array (analogous to a list in Python), they have a length attribute...
myParagraphElms.length;
which tells us how many paragraphs elements we were able to collect in the second example above.

But their return is not a real array! Instead, the return is actually a DOM object—a collection or formally a NodeList—not a JavaScript array.

Therefore you can’t slice() or push() or pop() or otherwise use JavaScript array methods on it; you have to iterate over it using a for loop and manually pack it into an array before it can be used that way. And once you pack it into an array, it’s no longer a “live” collection of objects such that it dynamically reflects changes made to the DOM (see A collection is not an array for more).

MooTools’ $$()

So what if you want your cake (an array) and you want to eat it to (a “live” collection of nodes)? That’s where MooTools comes to the rescue with the double dollar function: $$(). In short, the double dollar function is meant for collections of elements, and packs them into a live array. (If it helps keep $() and $$() straight, think single dollar equals single element, double dollar equals multiple elements.)

So how do we use the $$() method? We pass it unlimited CSS selectors:
// get all elements with class name
// “some-class” on the page
$$('.some-class');
// get all elements with any of these three classes (note
//  the commas)
$$('.some-class, .some-other-class, .some-other-class-still');
// inside all divs with “div-class” get all paragraphs with
// “p-class” (note the space)
$$('div.div-class p.p-class');
In the last example above, we use a tag and space to drill down; we can also drill down into a single object.

Let’s say we have a wrapper element that has the CSS id “myWrapper.” We want to use this myWrapper element as our starting point to drill down and gather all the elements inside of it that have the class name “remove.”

With MooTools, there are two ways we can do this:
// no CSS selector used with $(), but CSS "." selector for its
// getElements() method
var removeElms = $('myWrapper').getElements('.remove');
// CSS selectors for both id "#" and class "." (note the
// space)
var removeElms = $$('#myWrapper .remove');
MooTools not only enables us to grab elements using CSS selectors (tag names, class dots, or id pounds along with spaces and commas for the most common ones), it also enables us to set styles on an element.

In native JavaScript, if we wanted to set the style font-weight for an element, we would use:
// native Javascript: fontWeight
myDivElement.style.fontWeight = 'bold';
Notice that while CSS uses hyphens, JavaScript requires that hyphens be translated into camelCase; thus font-weight equals fontWeight and won’t work without this translation.

With MooTools, however, we can set an element style easily without having to do any translations:
// MooTools: standard font-weight
myDivElement.setStyle('font-weight', 'bold');

MooTools’ each()

Let’s go back to MooTools’ element arrays, such as the removeElms collection demonstrated a few examples ago. For most tasks, rather than writing a somewhat cumbersome for loop, we can iterate over an array objects using each().

MooTools will automatically return to us both the current element being iterated as well as the index of that particular iteration as the first and second parameters (respectively):
removeElms.each(function(myElm, index) {
    if (index === 0) {
        // if it's the first remove, hide it
        myElm.hide();
    } else {
        // else draw a red border around it
        myElm.setStyle('border', 'solid 1px red');
    }
});
Let’s take a quick look at what’s going on here: Element.each() actually takes just one parameter, and that parameter happens to be a function:
function(myElm, index) {
    // blah blah
}
Because we pass it a function, that is why it has the somewhat strange looking syntax ending in });: the curly bracket is the end of the function and the parenthesis the end of each().

There are a number of methods that MooTools provides just for working with the Element object alone, but this guide should give you the very basics of how HTML, CSS, and JavaScript—particularly through the MooTools framework—allows us to interact with elements. For complete documentation on the MooTools Element methods, see the MooTools Docs.

An Input Element’s "name" is to HTML what a Database Field is to MySQL

We now know a bit about the DOM and how JavaScript can interact with it; perhaps the single most important thing we do with the DOM is collect user-entered data. The database would not have any records in it if it were not for this ability, and it is helpful to understand how form elements map to the database.

While JavaScript or Python either one can rename keys, in a perfect world, the relationship between the DOM and the database would be one-to-one:
<form id="my-form">
    Password: <input type="password" name="pass" /><br />
    Name: <input type="text" name="name" /><br />
    Age: <input type="text" name="age" /><br />
    <input type="submit" />
</form>
Above are four input elements in a <form> element that look like so on an unstyled HTML page:


toObject()

[Source code here or here.]

In the IWS sharedLibrary.js, there are two very useful methods. One is called fromJSON() and the other is called toObject() (toObject is actually a wrapper for toJSON).

All we really need is a wrapper element of any sort surrounding a collection of inner inputs to make it work; we often do not even use the traditional <form> element in BriteCore for this purpose. If we use toObject(), it will provide us with what the Python recognizes as its js_dict parameter:
var submitObj = $('my-form').toObject();
The output of the above might be something like this (assuming the values to these object/dictionary keys are the ones the user actually typed in our form):
{
    'pass': 'a9(_}c8d7bic',
    'name': 'Cindy Sutterfield',
    'age': '19'
}
You can see the one-to-one correspondence between the screen elements and what ultimately becomes the js_dict that gets packed into the database.

fromJSON()

[Source code here or here.]

Likewise, when the JavaScript makes an Ajax request, Python assembles the dictionary to return, and json.dumps() (stringfies) it back to the front, where MooTools automatically converts this JSON string into a real JavaScript object. Using the same form element ($('my-form')) or other wrapper, we can pre-populate the values using fromJSON():
var submitObj = {
    'foo': 'bar'
};

var myRequest = new Request.JSON({
    url: '/urls.py?file=foo&method=bar',
    onSuccess: function(responseJSON) {
        // the JSON response populates the inputs
        // inside the form wrapper
        $('my-form').fromJSON(responseJSON);
    }
}).send(JSON.encode(submitObj));
The result?


JSON.decode() equals json.loads(); JSON.encode() equals json.dumps()

The last thing to notice is the send() part of the request: we use the MooTools’ method JSON.encode() to stringify the JSON. An equivalent IWS translation to the JSON methods for stringifying/de-stringifying includes:
// Javascript
JSON.decode(); // turn JSON string into JavaScript object
JSON.encode(); // stringify JavaScript object into JSON

// Python
import json
json.loads() // turn JSON string into Python dict
json.dumps() // stringify Python dict into JSON

A Magical Event

Last but not least, let’s take a look at JavaScript events, the magical ability JavaScript has of responding to mouse clicks, keyboard presses, or screen touches.

An event is quite literally an elaborate object (
dictionary
) that the browser fires off when a user interaction takes place; it contains a variety of information including the element that the event took place on as well as its x/y coordinates.

As long as JavaScript has a listener in place on an appropriate element for the appropriate type of event, it can respond to the firing of these events and act accordingly.

At IWS, we typically use an approach to events called “event delegation.” Rather than adding, say, 25 listeners to 25 different elements on the page, we instead use one listener on a wrapper element and “delegate” the event using a series of if/else if statements. We can do that because most events “bubble.”

Event Bubbling: Leave “on” off in MooTools

There are a wide range of DOM events (see here for a complete list), and they all begin with the word “on.” MooTools, however, always leaves off the “on" part, so when you pass the string name of the event to removeEvents and addEvent (demonstrated below), remember to leave it off!

But what is event bubbling?

When we looked at HTML earlier, we saw that elements were nested inside elements that were nested inside other elements.

An element that wraps other elements is a “parent” element and any elements that are nested inside of it are its “children.”

When a user clicks on one of these “child” elements, the event first registers on the child, then on that child’s parent, then on that parent’s parent, and so on until it bubbles all the way up to the top of the document.

If we attach an event listener to one of the parents, we can “listen” as its children bubble up, testing to see if a particular child we are interested in has bubbled to the top. If so, we take some course of action: a child with the CSS class “remove,” for example, might be an indication that we need to pop up a user dialog that a record is about to be deleted, requiring the user to confirm that this action was done on purpose.
// MooTools
// MooTools passes us back the event, here as evt
myParent.removeEvents('click').addEvent('click', function(evt) {

    var self = this;
    // MooTools offers us the element that was clicked via
    // “target.” We wrap it in the dollar function for IE
    var target = $(evt.target);
    // to make sure that anything else we do to the
    // element using MooTools’ methods will work on IE

    // we now run a series of tests on the element that
    // may involve tests for its tag, id, a class it
    // contains, or any other property we know is
    // true of a given child element we’re interested in

    if (target.id == 'save') {
        self.save();
    } else if (target.hasClass('next')) {
        self.goToNextScreen();
    } else if (target.hasClass('remove')) {
        // perform logic that pops up user confirmation--
        // if user OKs, fire off an Ajax call to remove a
        // record from the db
    }
});
Normally we further divide the logic up with an attachEvents method in a class that in turn calls a delegateClick method that contains the if / else if logic inside it, but the idea remains the same: attach one event listener to a parent element and listen for the individual children we’re interested in. Hence, “event delegation.”

Additionally, note that we use removeEvents before using addEvent: this keeps an event from doubling up.

If more than one event listener of a given type gets attached to the same element, each one will fire, leading to unpredictable behavior. The user clicks once, but because three instances of a “click” event listener were bound to that element, it is as though the user clicked three times in a row rather than just once.

For more on the Event object in MooTools, see Type: Event in the docs.

To attach an event using only native JavaScript, we could use the following. (I do not know how to remove an event this way, but—so far as I know at least—stacking shouldn’t occur, as we’re making an assignment that gets overwritten each time.)
// native Javascript
myParent.onclick = function(evnt) {
    // MooTools takes the following pain out of the equation:

    // if IE, the event will be bound on the global “window”
    // object rather than passed via “evnt”
    var evt = (evnt) ? evnt : window.event;
    // if IE, the target will be “srcElement”; if anybody
    // else, it will be “target”
    var target = (evt.srcElement) ? evt.srcElement : evt.target;

    if (target.id == 'save') {
        self.save();
    }
    // blah blah
};

Not All DOM Events Bubble!

Not all DOM events bubble. As per the DOM specification, the events onfocus and onblur (when the focus leaves an element) ''should not'' bubble, whereas onchange, onsubmit and onreset ''should'' bubble. You can also listen for onkeyup, onkeydown or onkeypress as these events bubble up the DOM on all browsers.

Not All Browsers Bubble Equally!

Now for a really big “gotcha”: remember earlier how we said that not all browser vendors were created equal? Guess what? Just to make life difficult for you, onchange, onsubmit, and onreset do not bubble up in IE even though they are supposed to! This is true of IE8 and below—I’m not sure what the official status is on IE9 and above.

To complicate matters further, to my knowledge, there is no work-around in MooTools, so if you are listening for these events and you want to support IE, you need to remember NOT to use event delegation but instead attach an event to each element you want to listen for.
// Ugly $$() hack we use inside RecordSet.Table.LineItemTable.attachEvents() in builder.js
if (Browser.ie) { // change events don't bubble in IE!
    $$('.linItm_manualCheckbox, .linItm_deductible, .rISWrap_options, .contentRow_amountInput, .lossExposureGroup, .rIIWrap_input, .singleLineAnswer, .multiLineAnswer, .listAnswer, .ifAddressQuestion_addressLine1, .ifAddressQuestion_addressLine2, input[name=city], select[name=state], input[name=zip], .scheduledItemInput, .contentRow_amountInput, .pb_classification').removeEvents('change').addEvent('change', function(evt) {
        if (self.localVars.hasFired !== 1) { // hack to keep IE from firing change event twice in a row
            self.localVars.hasFired = 1;
            self.delegateChange(evt);
        }
    });
} else {
    $(self.options.tableElements.body).removeEvents('change').addEvent('change', function(evt) {
        self.delegateChange(evt);
    });
}

Tuesday, July 05, 2011

Strikethrough in Gmail: At Long Last! (For Chrome and Firefox)


Gmail has come a long ways since I first started using it. However, one feature that is conspicuously absent is strikethrough text, which I often want to use because I'm proofreading someone else's writing and want to show the parts that I recommend deleting.

Now there is a way that I can have my cake and eat it too. If you're using Google Chrome, just click the Install button at Gmail Strikethrough Button. If you are using Firefox, first install the add-on Greasemonkey (if it isn't already installed), restart, then click Install.

Monday, July 04, 2011

JavaScript "Interactive Shell" (like Python) and Variable Hoisting


When I first started working at IWS, there were two teams, the front-end or "Interface" team and the back-end or "Systems" team. I was part of the Interface team writing JavaScript; the Systems team wrote Python. We have now gone to a new approach in development, the two teams merging into a single "Engineering" team. Though most of us have written at least some code in both languages, we've been knee-deep in cross-training.

As part of our cross-training, the new Engineering team has been further divided into "squadrons" of three with team members consisting of both (previous) Python and JavaScript developers working together on small projects. My team includes two Python developers who know relatively little about JavaScript, so the responsibility of catching them up to speed has fallen on my shoulders.

In working with my two teammates, I mentioned the concept of "hoisting" where variables are lifted to the top of function blocks by the interpreter. I also shared with them the PyCon video PyCon 2011: Javascript For People Who Know Python which offers a decent overview of JavaScript for the Python-minded.

The PyCon video suggested two things I want to mention: first, it suggested that JavaScript's behavior has an undeserved reputation for being unpredictable, and second, in the Q&A there was a question asked about an interactive shell for JavaScript. I'll come back to the former in a moment; regarding the latter, the video didn't mention it, but did you know Firebug provides its own interactive shell?

To use Firebug's "command editor" (as it is called), click the arrow next to the Console tab, then select Command Editor from the context menu:

The command editor offers a side window on the right; simply type or paste raw JavaScript and click Run to see the results in the console:

If you don't use Firefox, the MooTools' team has created the excellent JSFiddle, which comes with virtually all the major JS frameworks on board (not just MooTools) and which can be embedded directly in web pages and/or created examples can be saved and linked to: very cool stuff.

Okay, so now that we have an interaction shell, a large step toward understanding the "unpredictable" nature of JavaScript is to understand fully how hoisting works. I've been writing JavaScript for years now, and I'll have to say that if I was given an exam with the two examples listed at the beginning of JavaScript Scoping and Hoisting, I would have failed miserably.

The reason I would have failed is because JavaScript is normally backward to Python or PHP. Defining a private (definition scoped) variable in Python involves nothing more than declaring it in an assignment. If you want it to be global, you have to explicitly use the global keyword, exactly as you do in PHP (except in PHP, a semi-colon is mandatory).

But in JavaScript, if you want a private variable scoped to the function, you have to use the var keyword. If you omit the var keyword, the variable becomes available globally—define a variable without the var keyword anywhere in a script, and it's instantly global. (That said, if you declare a variable using the var keyword outside any functions or classes, it is also global—we might say that it is "private to the global scope" because though it was declared with the var keyword, it was nevertheless defined globally, outside a containing function or class.)

With this var / no var knowledge in place, that's why I would have failed an exam with the two examples from the article above: I did not understand exactly how hoisting worked, and this articles does an excellent job of taking another level of unpredictability out of JavaScript's run-time behavior.

So, check out JavaScript Scoping and Hoisting and put the claims to the test: fire up Firebug's command editor or try out JSFiddle and verify for yourself that the results accord with the claims made, then read the rest of the article to see why the results are counter-intuitive, and, ultimately, highly predictable once you understand what's going on under the hood.

Saturday, May 21, 2011

Balabolka: A Free Text-To-Speech (TTS) Swiss Army Knife for Windows Operating Systems


Let Your Computer Read to You

Typically when I'm home, I've set the company Mac aside and fallen back on a wide-screen Windows PC. I've gotten lazy as I've gotten older: I rather like having the computer read things to me I don't necessarily feel like reading myself. :)

Being read to is especially enjoyable when I'm either working on another task or I want to completely relax without having my eyes glued to the screen.

Enter Balabolka (Russian for "chatterer"), the free Windows text-to-speech (TTS) engine.

Balabolka is easy to use, and works with any TTS voice currently installed on your system (my current preference being "VW Kate"), and the documentation points you to additional voices you can download free of charge.

It supports a range of reading options including text-based documents of assorted types, pasted text, or the contents of the clipboard.

Balabolka has all the features you'd expect without getting carried off to bloat-land. Like most quality software, a lot of the best things about it are its most basic.

Two bread-and-butter features I needed that work especially well include the ability to save MP3s* and to "train" the dictionary to correct mispronunciations.
* In addition to MP3, Balabolka supports seven other file export formats: WAV, OGG, WMA, MP4, M4A, M4B, and AWB.

It also comes with the ability to save ID3 meta information to MP3s (though since I've been using Balabolka to record chapters in audio books, I prefer to bulk edit this info using another first-rate freeware program called Tag Scanner featured in Best Free MP3 Tag Editor).

Other useful features are the ability to have multiple documents open displayed as tabs along the top or bottom (your choice set in Options -> Settings) and the ability to use context menus while Balabolka is reading (both pictured in the screen shot shown above).

Balabolka is an all-around useful program that works well and is easy to use, and it comes in a portable version as well if you want to stick it on a USB drive and go. Just grab yourself a free copy today. Cheers! :)

Saturday, February 26, 2011

How to Minify HTML using PHP and Minify (including final HTML output of PHP files)


Full disclaimer: I did not actually implement this method in the project where I had it set up. I explain why below.

Before Doing Anything Else, Gzip Your Pages!

The first line of defense in reducing the payload of an HTML page is to gzip, or compress, your files. Make sure you have taken that step first if you haven't before doing anything else mentioned here, as that will provide the biggest pay-off.

Benchmark Your Speed, Before and After

I said I didn't implement this method on the particular project I was working on: often after your pages are gzipped, the benefits you get from further compression are either negligible or actually decrease load speed, the very thing you're hoping to improve.

Therefore, testing the speed of your pages before and after minification is advised: there are several free, online solutions available, and I happened to be using Webwait, which I liked because it is simple, does not require registration, and does not place restrictions.

In my case, results revealed that I was actually suffering a slight penalty in load time when stripping whitespace. That said, I have a writing site that has massive HTML pages, and when I have time, I will likely try it again there: I suspect that for very large files, minification by PHP before page load would provide a benefit.

Enough preamble.

Whitespace Removal: Basic Steps

There are several steps you need to take to set it up and get it working.

PHP Check:

First, is PHP already being used to render your page? If so, you're good to go as is—skip this paragraph. If not, does the page have an "htm," "html", or "shtml" file extension and does that extension matter? If the answer to the second question is no, just change your file extension to "php" and if your web service provides PHP you should be good. If the file extension does matter and you're hosted on an Apache server, take a look at Step 4 here on how to give PHP control of rendering your pages.

Note: If your extension is "shtml" and you currently make full use of server-side includes (SSI), you will need to reconfigure your pages to use PHP for the same task or else find a different solution. The beauty is, PHP can do everything Apache's support of SSI can and more.

Basic PHP Function with Associated "Gotchas" Explained:

Now we need to write a PHP function, so let's create a new file called compress_html.php that we include on all our pages we want to minify. Open the IDE or text editor of your choice and create this new file.

Now let's type a simple function.
<?php
// This function is probably too simple to use: It can bite.
function replace_tabs_newlines($content) {
    return preg_replace('(\r|\n|\t)', '', $content);
}
?>
A very simple function like this that strips newline characters and tabs might be all you'd need. The danger with this approach though happens when your site includes any inline JavaScript or CDATA that uses single-line comments (double slashes) or any pre-filled textareas that need to have newline characters for human readability. For example...
<script type="text/javascript">
// click event
myFormElement.onclick = function() {
     this.submit();
};
</script>
...our simple PHP function would cause all the JavaScript below the "click event" comment to fail, because it's just stripping newlines and tabs from the entire HTML page indiscriminately. Once the document gets parsed by the browser, as far as the interpreter is concerned, without the newlines the script tag contains nothing but a big, long comment: // click eventmyFormElement.onclick = function() {this.submit();};. That means, of course, that your click event just died a horrid death.

And if you have any textareas on your page with text already included in them, you'll also see just one big, long string. Probably not what you want.

Wrapper Function Using Minify:

My approach was to use Minify's little-documented and still-experimental HTML minifier instead. Simply download Minify, and check out this page on getting it set up and running.

Once Minify's in place, here's the new and improved, as before assumed to be saved to a file named compress_html.php; there's still nothing exotic about our now wrapper function, and if you know PHP, you could probably write a better one (and if you do, please post it here and share with the rest of us), but this one will at least get your foot through the door.
<?php
function replace_tabs_newlines($content) {
    require_once 'min/lib/Minify/HTML.php';
    require_once 'min/lib/Minify/CSS.php';
    require_once 'min/lib/JSMin.php';
    $content = Minify_HTML::minify($content, array(
        'cssMinifier' => array('Minify_CSS', 'minify'),
        'jsMinifier' => array('JSMin', 'minify')
    ));
    return $content;
}
?>
With some sort of callback method in place (such as the one above using Minify), you've now ensured that JavaScript and CSS at least should get handled correctly.

Serving Minified HTML on a Platter:

Now, simply include in the very top line of the PHP or HTML page you want to minify the following code, which includes the file that contains our replace_tabs_newlines function and calls ob_start:
<?php
// includes are relative to the including file: this example
// assumes compress_html.php lives at the same directory level
// as the file in which you'll include these lines
include('compress_html.php'); 
ob_start('replace_tabs_newlines');
?>
ob_start tells PHP to output any file contents to its internal memory buffer rather than the screen (think "o.utput b.uffering start"—you are turning on output buffering). The "replace_tabs_newlines" tells ob_start to send the raw output from the buffer through the function we just wrote and included in a separate file. Notice the quotation marks—single or double doesn't matter: ob_start expects the function reference to be a string.

Then, at the very bottom of the file after all other code and markup, include the following:
<?php
ob_end_flush();
?>
This does just what it says: it ends, or turns back off, output buffering, then flushes the contents previously held in the buffer either to the screen (if ob_start is passed no callback method) or through "replace_tabs_newlines" (or any other callback method, if one is specified).

Final Check:

View your source code, and you should see that white space and newlines * were stripped before the page was rendered.

That's it! Make sure to test your page speed before and after: Webwait or however. If you are seeing any performance increase, go for it! :)

Update on Getting a Totally Blank Output Page

I host through 1&1, and ran into a caveat when testing this on my writing site (which incidentally did improve speed performance for the large pages): I had to edit the file min/lib/Minify/CSS.php, taking out the three references to Minify in the require_once statements:
// edits made to "min/lib/Minify/CSS.php"

public static function minify($css, $options = array()) 
    {
        require_once 'CSS/Compressor.php'; // removed "Minify/" from the front
        if (isset($options['preserveComments']) 
            && !$options['preserveComments']) {
            $css = Minify_CSS_Compressor::process($css, $options);
        } else {
            require_once 'CommentPreserver.php'; // removed "Minify/" from the front
            $css = Minify_CommentPreserver::process(
                $css
                ,array('Minify_CSS_Compressor', 'process')
                ,array($options)
            );
        }
        if (! isset($options['currentDir']) && ! isset($options['prependRelativePath'])) {
            return $css;
        }
        require_once 'CSS/UriRewriter.php'; // removed "Minify/" from the front
        if (isset($options['currentDir'])) {
            return Minify_CSS_UriRewriter::rewrite(
                $css
                ,$options['currentDir']
                ,isset($options['docRoot']) ? $options['docRoot'] : $_SERVER['DOCUMENT_ROOT']
                ,isset($options['symlinks']) ? $options['symlinks'] : array()
            );  
        } else {
            return Minify_CSS_UriRewriter::prepend(
                $css
                ,$options['prependRelativePath']
            );
        }
    }
Note that (a) I am using the Minify library a directory level deeper (/some_dir/min), and (b) I only ran into problems when including it in files that lived in directories deeper than root, not at the root level itself.

However, I suspect that even if (a) were not an issue, (b) would probably still apply, and since CSS.php physically lives in the Minify directory, there should be no reason that directory should have to be specified at the front of the require_once path anyway, as includes are relative to the including file.

I can also verify that modifying this file in this way works equally well in root as it does any number of directory levels deeper.

* Minify does not necessarily strip all newlines depending on its heuristics and your particular page. If you can be assured that stripping all newlines and tabs will have no effect on your particular setup, you can replace "return $content;" with the return found in the otherwise too-simple first example. Minify will have safely stripped all comments from the CSS and JS and you'll be stripping any whitespace it "missed," but you've still got any pre-populated textareas to consider, inputs Minify knows to leave well enough alone.