|
|
<!doctype html>
<title>CodeMirror: Internals</title><meta charset="utf-8"/><link rel=stylesheet href="docs.css"><style>dl dl {margin: 0;} .update {color: #d40 !important}</style><script src="activebookmark.js"></script>
<div id=nav> <a href="https://codemirror.net/5"><h1>CodeMirror</h1><img id=logo src="logo.png"></a>
<ul> <li><a href="../index.html">Home</a> <li><a href="manual.html">Manual</a> <li><a href="https://github.com/codemirror/codemirror5">Code</a> </ul> <ul> <li><a href="#top">Introduction</a></li> <li><a href="#approach">General Approach</a></li> <li><a href="#input">Input</a></li> <li><a href="#selection">Selection</a></li> <li><a href="#update">Intelligent Updating</a></li> <li><a href="#parse">Parsing</a></li> <li><a href="#summary">What Gives?</a></li> <li><a href="#btree">Content Representation</a></li> <li><a href="#keymap">Key Maps</a></li> </ul></div>
<article>
<h2 id=top>(Re-) Implementing A Syntax-Highlighting Editor in JavaScript</h2>
<p style="font-size: 85%" id="intro"> <strong>Topic:</strong> JavaScript, code editor implementation<br> <strong>Author:</strong> Marijn Haverbeke<br> <strong>Date:</strong> March 2nd 2011 (updated November 13th 2011)</p>
<p style="padding: 0 3em 0 2em"><strong>Caution</strong>: this text was written briefly afterversion 2 was initially written. It no longer (even including theupdate at the bottom) fully represents the current implementation. I'mleaving it here as a historic document. For more up-to-dateinformation, look at the entriestagged <a href="http://marijnhaverbeke.nl/blog/#cm-internals">cm-internals</a>on my blog.</p>
<p>This is a followup tomy <a href="https://codemirror.net/5/story.html">Brutal Odyssey to theDark Side of the DOM Tree</a> story. That one describes themind-bending process of implementing (what would become) CodeMirror 1.This one describes the internals of CodeMirror 2, a complete rewriteand rethink of the old code base. I wanted to give this piece anotherHunter Thompson copycat subtitle, but somehow that would be out ofplace—the process this time around was one of straightforwardengineering, requiring no serious mind-bending whatsoever.</p>
<p>So, what is wrong with CodeMirror 1? I'd estimate, by mailing listactivity and general search-engine presence, that it has beenintegrated into about a thousand systems by now. The most prominentone, since a few weeks,being <a href="http://googlecode.blogspot.com/2011/01/make-quick-fixes-quicker-on-google.html">Googlecode's project hosting</a>. It works, and it's being used widely.</p>
<p>Still, I did not start replacing it because I was bored. CodeMirror1 was heavily reliant on <code>designMode</code>or <code>contentEditable</code> (depending on the browser). Neither ofthese are well specified (HTML5 triesto <a href="http://www.w3.org/TR/html5/editing.html#contenteditable">specify</a>their basics), and, more importantly, they tend to be one of the moreobscure and buggy areas of browser functionality—CodeMirror, by usingthis functionality in a non-typical way, was constantly running upagainst browser bugs. WebKit wouldn't show an empty line at the end ofthe document, and in some releases would suddenly get unbearably slow.Firefox would show the cursor in the wrong place. Internet Explorerwould insist on linkifying everything that looked like a URL or emailaddress, a behaviour that can't be turned off. Some bugs I managed towork around (which was often a frustrating, painful process), others,such as the Firefox cursor placement, I gave up on, and had to telluser after user that they were known problems, but not something Icould help.</p>
<p>Also, there is the fact that <code>designMode</code> (which seemedto be less buggy than <code>contentEditable</code> in Webkit andFirefox, and was thus used by CodeMirror 1 in those browsers) requiresa frame. Frames are another tricky area. It takes some effort toprevent getting tripped up by domain restrictions, they don'tinitialize synchronously, behave strangely in response to the backbutton, and, on several browsers, can't be moved around the DOMwithout having them re-initialize. They did provide a very nice way tonamespace the library, though—CodeMirror 1 could freely pollute thenamespace inside the frame.</p>
<p>Finally, working with an editable document means working withselection in arbitrary DOM structures. Internet Explorer (8 andbefore) has an utterly different (and awkward) selection API than allof the other browsers, and even among the different implementations of<code>document.selection</code>, details about how exactly a selectionis represented vary quite a bit. Add to that the fact that Opera'sselection support tended to be very buggy until recently, and you canimagine why CodeMirror 1 contains 700 lines of selection-handlingcode.</p>
<p>And that brings us to the main issue with the CodeMirror 1code base: The proportion of browser-bug-workarounds to realapplication code was getting dangerously high. By building on top of afew dodgy features, I put the system in a vulnerable position—anyincompatibility and bugginess in these features, I had to paper overwith my own code. Not only did I have to do some serious stunt-work toget it to work on older browsers (as detailed in theprevious <a href="https://codemirror.net/5/story.html">story</a>), thingsalso kept breaking in newly released versions, requiring me to come upwith <em>new</em> scary hacks in order to keep up. This was startingto lose its appeal.</p>
<section id=approach> <h2>General Approach</h2>
<p>What CodeMirror 2 does is try to sidestep most of the hairy hacksthat came up in version 1. I owe a lot to the<a href="http://ace.ajax.org">ACE</a> editor for inspiration on how toapproach this.</p>
<p>I absolutely did not want to be completely reliant on key events togenerate my input. Every JavaScript programmer knows that key eventinformation is horrible and incomplete. Some people (most awesomelyMihai Bazon with <a href="http://ymacs.org">Ymacs</a>) have been ableto build more or less functioning editors by directly reading keyevents, but it takes a lot of work (the kind of never-ending, fragilework I described earlier), and will never be able to properly supportthings like multi-keystoke international characterinput. <a href="#keymap" class="update">[see below for caveat]</a></p>
<p>So what I do is focus a hidden textarea, and let the browserbelieve that the user is typing into that. What we show to the user isa DOM structure we built to represent his document. If this is updatedquickly enough, and shows some kind of believable cursor, it feelslike a real text-input control.</p>
<p>Another big win is that this DOM representation does not have tospan the whole document. Some CodeMirror 1 users insisted that theyneeded to put a 30 thousand line XML document into CodeMirror. Puttingall that into the DOM takes a while, especially since, for somereason, an editable DOM tree is slower than a normal one on mostbrowsers. If we have full control over what we show, we must onlyensure that the visible part of the document has been added, and cando the rest only when needed. (Fortunately, the <code>onscroll</code>event works almost the same on all browsers, and lends itself well todisplaying things only as they are scrolled into view.)</p></section><section id="input"> <h2>Input</h2>
<p>ACE uses its hidden textarea only as a text input shim, and doesall cursor movement and things like text deletion itself by directlyhandling key events. CodeMirror's way is to let the browser do itsthing as much as possible, and not, for example, define its own set ofkey bindings. One way to do this would have been to have the wholedocument inside the hidden textarea, and after each key event updatethe display DOM to reflect what's in that textarea.</p>
<p>That'd be simple, but it is not realistic. For even medium-sizeddocument the editor would be constantly munging huge strings, and getterribly slow. What CodeMirror 2 does is put the current selection,along with an extra line on the top and on the bottom, into thetextarea.</p>
<p>This means that the arrow keys (and their ctrl-variations), home,end, etcetera, do not have to be handled specially. We just read thecursor position in the textarea, and update our cursor to match it.Also, copy and paste work pretty much for free, and people get theirnative key bindings, without any special work on my part. For example,I have emacs key bindings configured for Chrome and Firefox. There isno way for a script to detect this. <a class="update"href="#keymap">[no longer the case]</a></p>
<p>Of course, since only a small part of the document sits in thetextarea, keys like page up and ctrl-end won't do the right thing.CodeMirror is catching those events and handling them itself.</p></section><section id="selection"> <h2>Selection</h2>
<p>Getting and setting the selection range of a textarea in modernbrowsers is trivial—you just use the <code>selectionStart</code>and <code>selectionEnd</code> properties. On IE you have to do someinsane stuff with temporary ranges and compensating for the fact thatmoving the selection by a 'character' will treat \r\n as a singlecharacter, but even there it is possible to build functions thatreliably set and get the selection range.</p>
<p>But consider this typical case: When I'm somewhere in my document,press shift, and press the up arrow, something gets selected. Then, ifI, still holding shift, press the up arrow again, the top of myselection is adjusted. The selection remembers where its <em>head</em>and its <em>anchor</em> are, and moves the head when we shift-move.This is a generally accepted property of selections, and done right byevery editing component built in the past twenty years.</p>
<p>But not something that the browser selection APIs expose.</p>
<p>Great. So when someone creates an 'upside-down' selection, the nexttime CodeMirror has to update the textarea, it'll re-create theselection as an 'upside-up' selection, with the anchor at the top, andthe next cursor motion will behave in an unexpected way—our secondup-arrow press in the example above will not do anything, since it isinterpreted in exactly the same way as the first.</p>
<p>No problem. We'll just, ehm, detect that the selection isupside-down (you can tell by the way it was created), and then, whenan upside-down selection is present, and a cursor-moving key ispressed in combination with shift, we quickly collapse the selectionin the textarea to its start, allow the key to take effect, and thencombine its new head with its old anchor to get the <em>real</em>selection.</p>
<p>In short, scary hacks could not be avoided entirely in CodeMirror2.</p>
<p>And, the observant reader might ask, how do you even know that akey combo is a cursor-moving combo, if you claim you support anynative key bindings? Well, we don't, but we can learn. The editorkeeps a set known cursor-movement combos (initialized to thepredictable defaults), and updates this set when it observes thatpressing a certain key had (only) the effect of moving the cursor.This, of course, doesn't work if the first time the key is used wasfor extending an inverted selection, but it works most of thetime.</p></section><section id="update"> <h2>Intelligent Updating</h2>
<p>One thing that always comes up when you have a complicated internalstate that's reflected in some user-visible external representation(in this case, the displayed code and the textarea's content) iskeeping the two in sync. The naive way is to just update the displayevery time you change your state, but this is not only error prone(you'll forget), it also easily leads to duplicate work on big,composite operations. Then you start passing around flags indicatingwhether the display should be updated in an attempt to be efficientagain and, well, at that point you might as well give up completely.</p>
<p>I did go down that road, but then switched to a much simpler model:simply keep track of all the things that have been changed during anaction, and then, only at the end, use this information to update theuser-visible display.</p>
<p>CodeMirror uses a concept of <em>operations</em>, which start bycalling a specific set-up function that clears the state and end bycalling another function that reads this state and does the requiredupdating. Most event handlers, and all the user-visible methods thatchange state are wrapped like this. There's a methodcalled <code>operation</code> that accepts a function, and returnsanother function that wraps the given function as an operation.</p>
<p>It's trivial to extend this (as CodeMirror does) to detect nesting,and, when an operation is started inside an operation, simplyincrement the nesting count, and only do the updating when this countreaches zero again.</p>
<p>If we have a set of changed ranges and know the currently shownrange, we can (with some awkward code to deal with the fact thatchanges can add and remove lines, so we're dealing with a changingcoordinate system) construct a map of the ranges that were leftintact. We can then compare this map with the part of the documentthat's currently visible (based on scroll offset and editor height) todetermine whether something needs to be updated.</p>
<p>CodeMirror uses two update algorithms—a full refresh, where it justdiscards the whole part of the DOM that contains the edited text andrebuilds it, and a patch algorithm, where it uses the informationabout changed and intact ranges to update only the out-of-date partsof the DOM. When more than 30 percent (which is the current heuristic,might change) of the lines need to be updated, the full refresh ischosen (since it's faster to do than painstakingly finding andupdating all the changed lines), in the other case it does thepatching (so that, if you scroll a line or select another character,the whole screen doesn't have to bere-rendered). <span class="update">[the full-refreshalgorithm was dropped, it wasn't really faster than the patchingone]</span></p>
<p>All updating uses <code>innerHTML</code> rather than direct DOMmanipulation, since that still seems to be by far the fastest way tobuild documents. There's a per-line function that combines thehighlighting, <a href="manual.html#markText">marking</a>, andselection info for that line into a snippet of HTML. The patch updateruses this to reset individual lines, the refresh updater builds anHTML chunk for the whole visible document at once, and then uses asingle <code>innerHTML</code> update to do the refresh.</p></section><section id="parse"> <h2>Parsers can be Simple</h2>
<p>When I wrote CodeMirror 1, Ithought <a href="https://codemirror.net/5/story.html#parser">interruptibleparsers</a> were a hugely scary and complicated thing, and I used abunch of heavyweight abstractions to keep this supposed complexityunder control: parserswere <a href="http://bob.pythonmac.org/archives/2005/07/06/iteration-in-javascript/">iterators</a>that consumed input from another iterator, and used funnyclosure-resetting tricks to copy and resume themselves.</p>
<p>This made for a rather nice system, in that parsers formed strictlyseparate modules, and could be composed in predictable ways.Unfortunately, it was quite slow (stacking three or four iterators ontop of each other), and extremely intimidating to people not used to afunctional programming style.</p>
<p>With a few small changes, however, we can keep all thoseadvantages, but simplify the API and make the whole thing lessindirect and inefficient. CodeMirror2's <a href="manual.html#modeapi">mode API</a> uses explicit stateobjects, and makes the parser/tokenizer a function that simply takes astate and a character stream abstraction, advances the stream onetoken, and returns the way the token should be styled. This state maybe copied, optionally in a mode-defined way, in order to be able tocontinue a parse at a given point. Even someone who's never touched alambda in his life can understand this approach. Additionally, farfewer objects are allocated in the course of parsing now.</p>
<p>The biggest speedup comes from the fact that the parsing no longerhas to touch the DOM though. In CodeMirror 1, on an older browser, youcould <em>see</em> the parser work its way through the document,managing some twenty lines in each 50-millisecond time slice it got. Itwas reading its input from the DOM, and updating the DOM as it wentalong, which any experienced JavaScript programmer will immediatelyspot as a recipe for slowness. In CodeMirror 2, the parser usuallyfinishes the whole document in a single 100-millisecond time slice—itmanages some 1500 lines during that time on Chrome. All it has to dois munge strings, so there is no real reason for it to be slowanymore.</p></section><section id="summary"> <h2>What Gives?</h2>
<p>Given all this, what can you expect from CodeMirror 2?</p>
<ul>
<li><strong>Small.</strong> the base library issome <span class="update">45k</span> when minifiednow, <span class="update">17k</span> when gzipped. It's smaller thanits own logo.</li>
<li><strong>Lightweight.</strong> CodeMirror 2 initializes veryquickly, and does almost no work when it is not focused. This meansyou can treat it almost like a textarea, have multiple instances on apage without trouble.</li>
<li><strong>Huge document support.</strong> Since highlighting isreally fast, and no DOM structure is being built for non-visiblecontent, you don't have to worry about locking up your browser when auser enters a megabyte-sized document.</li>
<li><strong>Extended API.</strong> Some things kept coming up in themailing list, such as marking pieces of text or lines, which wereextremely hard to do with CodeMirror 1. The new version has propersupport for these built in.</li>
<li><strong>Tab support.</strong> Tabs inside editable documents were,for some reason, a no-go. At least six different people announced theywere going to add tab support to CodeMirror 1, none survived (I mean,none delivered a working version). CodeMirror 2 no longer removes tabsfrom your document.</li>
<li><strong>Sane styling.</strong> <code>iframe</code> nodes aren'treally known for respecting document flow. Now that an editor instanceis a plain <code>div</code> element, it is much easier to size it tofit the surrounding elements. You don't even have to make it scroll ifyou do not <a href="../demo/resize.html">want to</a>.</li>
</ul>
<p>On the downside, a CodeMirror 2 instance is <em>not</em> a nativeeditable component. Though it does its best to emulate such acomponent as much as possible, there is functionality that browsersjust do not allow us to hook into. Doing select-all from the contextmenu, for example, is not currently detected by CodeMirror.</p>
<p id="changes" style="margin-top: 2em;"><span style="font-weight:bold">[Updates from November 13th 2011]</span> Recently, I've madesome changes to the codebase that cause some of the text above to nolonger be current. I've left the text intact, but added markers at thepassages that are now inaccurate. The new situation is describedbelow.</p></section><section id="btree"> <h2>Content Representation</h2>
<p>The original implementation of CodeMirror 2 represented thedocument as a flat array of line objects. This worked well—splicingarrays will require the part of the array after the splice to bemoved, but this is basically just a simple <code>memmove</code> of abunch of pointers, so it is cheap even for huge documents.</p>
<p>However, I recently added line wrapping and code folding (linecollapsing, basically). Once lines start taking up a non-constantamount of vertical space, looking up a line by vertical position(which is needed when someone clicks the document, and to determinethe visible part of the document during scrolling) can only be donewith a linear scan through the whole array, summing up line heights asyou go. Seeing how I've been going out of my way to make big documentsfast, this is not acceptable.</p>
<p>The new representation is based on a B-tree. The leaves of the treecontain arrays of line objects, with a fixed minimum and maximum size,and the non-leaf nodes simply hold arrays of child nodes. Each nodestores both the amount of lines that live below them and the verticalspace taken up by these lines. This allows the tree to be indexed bothby line number and by vertical position, and all access haslogarithmic complexity in relation to the document size.</p>
<p>I gave line objects and tree nodes parent pointers, to the nodeabove them. When a line has to update its height, it can simply walkthese pointers to the top of the tree, adding or subtracting thedifference in height from each node it encounters. The parent pointersalso make it cheaper (in complexity terms, the difference is probablytiny in normal-sized documents) to find the current line number whengiven a line object. In the old approach, the whole document array hadto be searched. Now, we can just walk up the tree and count the sizesof the nodes coming before us at each level.</p>
<p>I chose B-trees, not regular binary trees, mostly because theyallow for very fast bulk insertions and deletions. When there is a bigchange to a document, it typically involves adding, deleting, orreplacing a chunk of subsequent lines. In a regular balanced tree, allthese inserts or deletes would have to be done separately, which couldbe really expensive. In a B-tree, to insert a chunk, you just walkdown the tree once to find where it should go, insert them all in oneshot, and then break up the node if needed. This breaking up mightinvolve breaking up nodes further up, but only requires a single passback up the tree. For deletion, I'm somewhat lax in keeping thingsbalanced—I just collapse nodes into a leaf when their child count goesbelow a given number. This means that there are some weird editingpatterns that may result in a seriously unbalanced tree, but even suchan unbalanced tree will perform well, unless you spend a day makingstrangely repeating edits to a really big document.</p></section><section id="keymap"> <h2>Keymaps</h2>
<p><a href="#approach">Above</a>, I claimed that directly catching keyevents for things like cursor movement is impractical because itrequires some browser-specific kludges. I then proceeded to explainsome awful <a href="#selection">hacks</a> that were needed to make itpossible for the selection changes to be detected through thetextarea. In fact, the second hack is about as bad as the first.</p>
<p>On top of that, in the presence of user-configurable tab sizes andcollapsed and wrapped lines, lining up cursor movement in the textareawith what's visible on the screen becomes a nightmare. Thus, I'vedecided to move to a model where the textarea's selection is no longerdepended on.</p>
<p>So I moved to a model where all cursor movement is handled by myown code. This adds support for a goal column, proper interaction ofcursor movement with collapsed lines, and makes it possible forvertical movement to move through wrapped lines properly, instead ofjust treating them like non-wrapped lines.</p>
<p>The key event handlers now translate the key event into a string,something like <code>Ctrl-Home</code> or <code>Shift-Cmd-R</code>, anduse that string to look up an action to perform. To make keybindingcustomizable, this lookup goes througha <a href="manual.html#option_keyMap">table</a>, using a scheme thatallows such tables to be chained together (for example, the defaultMac bindings fall through to a table named 'emacsy', which definesbasic Emacs-style bindings like <code>Ctrl-F</code>, and which is alsoused by the custom Emacs bindings).</p>
<p>A newoption <a href="manual.html#option_extraKeys"><code>extraKeys</code></a>allows ad-hoc keybindings to be defined in a much nicer way than whatwas possible with theold <a href="manual.html#option_onKeyEvent"><code>onKeyEvent</code></a>callback. You simply provide an object mapping key identifiers tofunctions, instead of painstakingly looking at raw key events.</p>
<p>Built-in commands map to strings, rather than functions, forexample <code>"goLineUp"</code> is the default action bound to the uparrow key. This allows new keymaps to refer to them withoutduplicating any code. New commands can be defined by assigning tothe <code>CodeMirror.commands</code> object, which maps such commandsto functions.</p>
<p>The hidden textarea now only holds the current selection, with noextra characters around it. This has a nice advantage: polling forinput becomes much, much faster. If there's a big selection, this textdoes not have to be read from the textarea every time—when we poll,just noticing that something is still selected is enough to tell usthat no new text was typed.</p>
<p>The reason that cheap polling is important is that many browsers donot fire useful events on IME (input method engine) input, which isthe thing where people inputting a language like Japanese or Chineseuse multiple keystrokes to create a character or sequence ofcharacters. Most modern browsers fire <code>input</code> when thecomposing is finished, but many don't fire anything when the characteris updated <em>during</em> composition. So we poll, whenever theeditor is focused, to provide immediate updates of the display.</p>
</section></article>
|
