This article will help you understand the internals of JavaScript - even the weird parts. Every line of code that you write in JavaScript will make complete sense once you know how it has been interpreted by the underlying engine. You'll learn multiple ways of downloading scripts based on the use case, and how the parser generates an Abstract Syntax Tree and its heuristics while parsing the code. Let's dive deep into the internals of JavaScript engines - starting from downloading scripts.
JavaScript is one of the most popular languages today. Gone are the days when people would use JavaScript merely for handling DOM event listeners and for a few undemanding tasks. Today, you can build an entire application from the ground up using JavaScript. JavaScript has taken over the winds, lands and the seas. With Node.js invading the gamut of server-side technologies and the advent of rich and powerful client-side libraries and frameworks like React, Angular and Vue, JavaScript has conquered the web. Applications are shipping a lot of JavaScript over the wires. Almost all of the complicated tasks of an application are now implemented using JavaScript.
While this is all great, it is disheartening to see that most of these applications lack even the minimal user experience. We keep adding functionalities to our application without taking into effect its performance implications. It is important that we follow proper techniques to deliver optimized code.
In this series of tutorials, we’ll first understand what is wrong with the conventional techniques and then we’ll dig deeper to learn some of the techniques that’ll help us write optimized code. We’ll also understand how our code gets parsed, interpreted and compiled by the underlying JavaScript engine and what works best for our engines. While the syntax of JavaScript is pretty easy to grasp, understanding its internals is a more daunting task. We’ll start from the very basics and eventually take over the beast. Let’s get going.
Understanding the Script Tag
Let’s consider a simple HTML file:
<!DOCTYPE html><html><head><scriptsrc='./js/first.js'></script><scriptsrc='./js/second.js'></script><scriptsrc='./js/third.js'></script><scriptsrc='./js/fourth.js'></script></head><body><div>Understanding the script tag</div></body></html>
first.js includes the following code:
console.log('first.js file')
second.js includes the following code:
console.log('second.js file')
I’ve set up an express server for demonstrating the concepts explained in the article. If you want to experiment along the way, please feel free to clone my GitHub repository.
Let’s see what happens when we open this HTML file in the browser:
![loading-scripts]( "loading-scripts")
The browser starts parsing the HTML code. When it comes across a script tag in the head section, the HTML parsing is paused. An HTTP request is sent to the server to fetch the script. The browser waits until the entire script is downloaded. It then does the work of parsing, interpreting and executing the downloaded script (we’ll get into the details of the entire process later in the article). This happens for each of the four scripts.
Once this is done, the browser resumes its work of parsing HTML and creating DOM nodes. The user, who is patiently staring at the screen waiting for something to load, doesn’t know most of his time is spent executing JavaScript code (even the code that may not be required during the startup). Script tags are blocking in nature. They block the rendering of the DOM. Your high school teacher might have told you, “Always put the script tags below body.” Now that you know script tags block rendering of the DOM, it makes sense to put them below the HTML. It is better to show non-interactive content (for a few milliseconds until the JavaScript code gets ready) than nothing at all.
Imagine you have a very big chain of DOM nodes — tens of thousands of them. According to what we’ve learned so far, in this case, the user would see a lot of content but he won’t be able to interact even with the tiniest piece. I’m sure you have visited websites that show you the entire content almost instantly but don’t let you scroll down or even click on any element. The page doesn’t seem to move for a few seconds. Isn’t that frustrating? The next obvious question is: when should we load the scripts — at the start before parsing of HTML or at the end after the HTML? Let’s analyze the problem a bit more.
Our end goal is clear — to load assets instantly during the startup. Our first approach of parsing scripts first and then the HTML renders a good user experience, but it eats up a lot of the user’s time by showing him blank screen while the content is getting executed. The problem with this approach is that it gets worse with an increase in the number of scripts since waiting time (load time) is directly proportional to the number of scripts. For every script, we make a ride to the server and wait until it gets downloaded.
Can we dump all of the JavaScript code in one file? This would reduce the number of rides we make to the server. That would mean dumping tens of thousands of lines of JavaScript into one file. I’m definitely not going for this. This would mean compromising with my code ethics.
Heard of Gulp, webpack? They are nothing but module bundlers in simple terms. Module bundlers, eh? You write your JavaScript code in any number of files (as many modules as you wish). Module bundlers bundle all of your JavaScript files and static assets in one big chunk, and you can simply add this one big file in your HTML.
Certainly, we reduced the number of HTTP requests to the server. Are we not still downloading, parsing and executing the entire content? Can we do something about it? There’s something called as code splitting. With webpack, you can split your code into different bundles. Dump all the common code in one bundle (like Vendor.js, which has all the common libraries to be used across the project) and others that are specific to modules.
For example, let’s say you are building an eCommerce website. You have different modules for Store, Transactions History and Payment. It doesn’t make sense to load your payment-specific code on the store-specific page. Bundlers have solved our problem by making fewer HTTP requests to the server.
Now, let’s consider one use case here. I’ve added Google Analytics to gain insights into how users are interacting with my eCommerce website. Google Analytics script is not required during the startup. We may want to load the app-specific stuff first and then other secondary scripts.
Downloading Scripts Asynchronously
When you add the async keyword in your script tag, the browser downloads that script asynchronously. The browser doesn’t pause the parsing of DOM when it comes across a script tag with async keyword. The script is downloaded in another thread without disturbing the main thread, and, once it is downloaded, the browser pauses the parsing of HTML and gets busy in parsing this script code. Once the parsing of this JavaScript code is completed, it is executed in another thread and the browser resumes its work of parsing HTML. We’ve saved the waiting time of the browser while the script is getting downloaded.
Let’s say we want to download two of our scripts asynchronously:
<!DOCTYPE html><html><head><scriptasyncsrc='./js/first.js'></script><scriptasyncsrc='./js/second.js'></script><scriptsrc='./js/third.js'></script><scriptsrc='./js/fourth.js'></script></head><body><div>Understanding the script tag</div></body></html>
Deferring the Execution of Scripts
When you add defer keyword in your script tag, the browser doesn’t execute that script until the HTML parsing is completed. Defer simply means the execution of the file is deferred or delayed. The script is downloaded in another thread and is executed only after the HTML parsing is completed.
<!DOCTYPE html><html><head><scriptdefersrc='./js/first.js'></script><scriptdefersrc='./js/second.js'></script><scriptsrc='./js/third.js'></script><scriptsrc='./js/fourth.js'></script></head><body><div>Understanding the script tag</div></body></html>
![defer-scripts]( "defer-scripts")
As we can see in the above screenshot, third.js and fourth.js were executed before first.js and second.js.
Here’s a brief overview of the three techniques of adding scripts:
![comparison]( "comparison")
Until now, we’ve understood how scripts are downloaded and what the most effective ways of downloading scripts are. Let’s understand what happens after a script is downloaded. (We’re considering Chrome browser, although almost all of the popular browsers follow similar steps.)
Chrome uses V8 as the underlying JavaScript Engine. It consists of the following components.
![js-engine]( "js-engine")
- Parser - JavaScript is fed into a Parser, which generates an Abstract Syntax Tree
- Interpreter - Abstract Syntax Tree is the input for the V8 Ignition Interpreter, which generates the ByteCode
- Compiler - The Turbofan Compiler of the V8 Engine takes in the ByteCode and generates machine code
- Optimizing Compiler - It takes ByteCode and some profiling data as the input and generates optimized machine code
We’ll get into the details of each of these components.
Parsing JavaScript Code
The JavaScript source code is first converted to tokens. Tokens represent the alphabet of a language. Every unit in the source code is identified by the grammar of the language that you’re using.
So, something like var a = 1 is a valid JavaScript statement. It can be broken down to tokens (‘var’, ‘a’, ‘=’, ‘1’) that match with the language grammar. However, something like variable a = 2 is not a valid JavaScript statement because its grammar doesn’t specify anything related to the variable keyword. Now, with the help of these tokens, the parser generates an Abstract Syntax Tree (AST) and scopes. AST, in simple terms, is a data structure that is used for representing the source code. Scopes are also data structures, used for identifying the scope of variables in their defined blocks. For example, a local variable would be accessible in the local scope and not in global scope. These constraints are defined in these scopes data structures.
Consider this simple JavaScript code snippet -
var a = 2
I refer AST Explorer to check the AST generated for my code. The AST for the above code would look something like this:
{"type":"Program","start":0,"end":9,"body":[{"type":"VariableDeclaration","start":0,"end":9,"declarations":[{"type":"VariableDeclarator","start":4,"end":9,"id":{"type":"Identifier","start":4,"end":5,"name":"a"},"init":{"type":"Literal","start":8,"end":9,"value":2,"raw":"2"}}],"kind":"var"}],"sourceType":"module"}
Let’s try to make sense of the above AST. It’s a JavaScript object with properties as type, start, end, body and sourceType. start is the index of the first character, and end is the length of your code, which is var a = 2 in this case. body contains the definition of the code. It’s an array with a single object since there is only one statement of the type VariableDeclaration in our program. Inside VariableDeclaration, it specifies the identifier a and its initial value as 2. Check id and init objects. The kind of declaration is var. It can also be let or const.
Let’s consider one more example to get better understanding of ASTs:
functionfoo(){let bar =2return bar
}
And its AST is as follows -
{"type":"Program","start":0,"end":50,"body":[{"type":"FunctionDeclaration","start":0,"end":50,"id":{"type":"Identifier","start":9,"end":12,"name":"foo"},"expression":false,"generator":false,"params":[],"body":{"type":"BlockStatement","start":16,"end":50,"body":[{"type":"VariableDeclaration","start":22,"end":33,"declarations":[{"type":"VariableDeclarator","start":26,"end":33,"id":{"type":"Identifier","start":26,"end":29,"name":"bar"},"init":{"type":"Literal","start":32,"end":33,"value":2,"raw":"2"}}],"kind":"let"},{"type":"ReturnStatement","start":38,"end":48,"argument":{"type":"Identifier","start":45,"end":48,"name":"bar"}}]}}],"sourceType":"module"}
Again, it has properties — type, start, end, body and sourceType. start is 0, which means the first character is at position 0, and end is 50, which means the length of the code is 50. body is an array with one object of the type FunctionDeclaration. The name of the function foo is specified in the id object. This function doesn’t take any arguments hence params is an empty array. The body of the FunctionDeclaration is of type BlockStatement. BlockStatement identifies the scope of the function. The body of the BlockStatement has two objects for VariableDeclaration and ReturnStatement. VariableDeclaration is same as we saw in the previous example. ReturnStatement contains an argument with name bar, as bar is being returned by the function foo.
This is it. This is how ASTs are generated. When I heard of ASTs the first time, I thought of them as big scary trees with complicated nodes. But now that we’ve got a good hold on what ASTs are, don’t you think they are just a group of nicely designed nodes representing the semantics of a program?
Parser also takes care of Scopes.
let globalVar =2functionfoo(){let globalVar =3
console.log('globalVar', globalVar)}
Function foo would print 3 and not 2 because the value of globalVar in its scope is 3. While parsing the JavaScript code, the parser generates its corresponding scopes as well.
When a globalVar is referred in function foo, we first look for globalVar in the functional scope. If that variable is not found in the functional scope, we look up to its parent, which in this case is the global object. Let’s consider one more example:
let globalVar =2functionfoo(){let localVar =3
console.log('localVar', localVar)
console.log('globalVar', globalVar)}
console.log('localVar', localVar)
console.log('globalVar', globalVar)
The console statements inside function foo would print 3 and 2 while the console statements outside function foo would print undefined and 3. This is because localVar is not accessible outside function foo. It is defined in the scope of function foo and so a lookup for localVar outside of it results in undefined.
Parsing in V8
V8 uses two parsers for parsing JavaScript code, called as Parser and Pre-Parser. To understand the need of two parsers, let’s consider the code below:
functionfoo(){
console.log('I\'m inside function foo')}functionbar(){
console.log('I\'m inside function bar')}foo()
When the above code gets parsed, the parser would generate an AST representing the function foo and function bar. However, the function bar is not called anywhere in the program. We’re spending time in parsing and compiling functions that are not used, at least during the startup. bar may be called at a later stage, maybe on click of a button. But it is clearly not needed during the startup. Can we save this time by not compiling function bar during the startup? Yes, we can!
Parser is what we’re doing till now. It parses all of your code, builds ASTs, scopes and finds all the syntax errors. The Pre-Parser is like a fast parser. It only compiles what is needed and skips over the functions that are not called. It builds scopes but doesn’t build an AST. It finds only a restricted set of errors and is approximately twice as fast as the Parser. V8 employs a heuristic approach to determine the parsing technique at runtime.
Let’s consider one example to understand how V8 parses JavaScript code:
(functionfoo(){
console.log('I\'m an IIFE function')functionbar(){
console.log('I\'m an inner function inside IIFE')}})()
When the parser comes across the opening parenthesis, it understands that this is an IIFE and it would be called immediately, so it parses the foo function using full parser or eager parser. Inside foo, when it comes across the function bar, it lazily parses or pre-parses the function bar because, based on its heuristics, it knows that the function bar won’t be called immediately. As the function foo is fully parsed, V8 builds its AST as well as scopes while it doesn’t build an AST for function bar. It builds only scopes for function bar.
Have you encountered this situation ever while writing JavaScript code:
![parser-error]( "parser-error")
The code throws an error only when you call the function fnClickListener. This is because V8 doesn’t parse this function on the first load. It parses the function fnClickListener only when you call it.
Let’s consider a few more examples to better understand the heuristics followed by V8.
functiontoBeCalled(){}toBeCalled()
The function toBeCalled is lazily parsed by the V8 engine. When it encounters the call to function toBeCalled, it now uses a full parser to parse it completely. The time spent in lazily parsing the function toBeCalled is actually wasted time. While V8 is lazily parsing function toBeCalled, it doesn’t know that the immediate statement would be a call to this function. To avoid this, you can tell V8 which functions are to be eagerly-parsed (fully-parsed).
(functiontoBeCalled(){})toBeCalled()
Wrapping a function in parentheses is an indicator to V8 that this function is to be eagerly-parsed. You can also add an exclamation mark before the function declaration to tell V8 to eagerly-parse that function.
!functiontoBeCalled(){}toBeCalled()
Parsing of Inner Functions
functionouter(){functioninner(){}}
In this case, V8 lazily parses both the functions, outer and inner. When we call outer, the outer function is eagerly/fully-parsed and inner function is again lazily parsed. This means inner function is lazily parsed twice. It gets even worse when functions are heavily nested.
functionouter(){functioninner(){functioninsideInner(){}}return inner
}
Initially, all the three functions outer, inner and insideInner are lazily parsed.
let innerFn =outer()innerFn()
When we call function outer, it is fully-parsed and functions inner and insideInner are lazily parsed. Now, when we call inner, inner is fully parsed and insideInner is lazily parsed. That makes insideInner get parsed thrice. Don’t use nested functions when they are not required. Use nested functions appropriately!
Parsing of Closures
(functionouter(){let a =2let b =3functioninner(){return a
}return inner
})
In the above code snippet, since the function outer is wrapped in parentheses, it is eagerly parsed. Function inner is lazily parsed. inner returns variable a, which is in the scope of its outer function. This is a valid case for closure.
let innerFn =outer()innerFn()
innerFn very well returns a value of 2 since it has access to variable a of its parent scope. While parsing the function inner, when V8 comes across the variable a, it looks up for variable a in the context of inner function. Since a is not present in the scope of inner, it checks it in the scope of function outer. V8 understands that the variable a is to be saved in the function context and is to be preserved even after outer function has completed its execution. So, variable a is stored in the function context of outer and is preserved until its dependent function inner has completed execution. Please note, variable b is not preserved in this case as it is not used in any of the inner functions.
When we call function innerFn, the value of a is not found in the call stack, we then look up for its value in the function context. Lookups in function context are costly as compared to lookups in the call stack.
Let’s check the parsed code generated by V8.
functionfnCalled(){
console.log('Inside fnCalled')}functionfnNotCalled(){
console.log('Inside fnNotCalled')}fnCalled()
As per our understanding, both of these functions will be lazily parsed and when we make a function call to fnCalled, it would be fully parsed and print Inside fnCalled. Let’s see this in action. Run the file containing the above code as node --trace_parse parse.js. If you’ve cloned my GitHub repository, you’ll find this file under public/js folder. parse.js is the name of the file, and --trace_parse serves as an indicator to the runtime of nodejs to print the parsed output. This command would generate a dump of parsing logs. I’ll save the output of this command in a file parsedOutput.txt. For now, all that makes sense is the below screenshot of the dump.
![parsed-output]( "parsed-output")
Function fnCalled is parsed, but function fnNotCalled is not parsed. Try searching for fnNotCalled in the dump.
Script Streaming
Now that we know how parsing works in V8, let’s understand one concept related to Script Streaming. Script Streaming is effective from Chrome version 41.
From what we’ve learned till now, we know it’s the main thread that parses the JavaScript code (even with async and defer keywords). With Script Streaming in place, now the parsing can happen in another thread. While the script is still getting downloaded by the main thread, the parser thread can start parsing the script. This means that the parsing would be completed in line with the download. This technique proves very helpful for large scripts and slow network connections. Check out the below image to understand how the browser operates with Script Streaming and without Script Streaming.
![streaming]( "streaming")
In this tutorial, we learned multiple ways of downloading scripts based on the use case. We learned how the parser generates an Abstract Syntax Tree and its heuristics while parsing the code. Later in the article, we learned about Script Streaming. In the next article, we’ll learn how parsing code gets compiled by the V8 compiler.
For More on Building Apps with jQuery:
Want to learn more about creating great user interfaces with jQuery? Check out
Kendo UI for jQuery - our complete UI component library that allows you to quickly build high-quality, responsive apps. It includes all the components you’ll need, from grids and charts to schedulers and dials.
