Tag Archives: Windows Insiders

ORTC API is now available in Microsoft Edge

In October last year, we announced our intention to implement ORTC in Microsoft Edge with a focus on video and audio communications, and we have been hard at work on our implementation since then. Today, we are excited to announce that a preview version of our ORTC API implementation is available in the latest Windows Insider Preview release.

ORTC Support in Microsoft Edge is the result of a close collaboration between Microsoft’s Operating Systems Group and Skype teams. Together we bring over 20 years of web platform experience and over 12 years of expertise in building one of the largest and most reliable real-time communications service for consumers and business users. Our goal is to enable developers around the world to build experiences that include the ability to talk to Skype users and other WebRTC compatible communication services.

We look forward to the web community building more user scenarios enabled by ORTC. In support of that, we would like to share with you what we have provided in this preview version of our ORTC implementation, and the basic steps in building simple 1:1 audio and video communications.

What we are providing

The below diagram is part of the Overview section in the ORTC API spec. It provides a very high level summary of the relationships between ORTC objects, and a useful illustration of a code flow, from capture media stream tracks as input to the RtpSender objects, and all the way to the media stream tracks out of the RtpReceiver objects that can be rendered to video/audio tags. We recommend using the diagram as a reference when you start learning ORTC API.

High level summary of the relationships between ORTC objects.

Click to enlarge

Our initial ORTC implementation includes the following components:

  1. ORTC API Support. Our primary focus right now is audio/video communications. We have implemented the following objects: IceGatherer, IceTransport, DtlsTransport, RtpSender, RtpReceiver, as well as the RTCStats interfaces that are not shown directly in the diagram.
  2. RTP/RTCP multiplexing is supported and is required for use with DtlsTransport. A/V multiplexing is also supported.
  3. STUN/TURN/ICE support. We support STUN (RFC 5389), TURN (RFC 5766) as well as ICE (RFC 5245). Within ICE, regular nomination is supported, with aggressive nomination partially supported (as a receiver). DTLS-SRTP (RFC 5764) is supported, based on DTLS 1.0 (RFC 4347).
  4. Codec support. For audio codecs, we support G.711, G.722, Opus and SILK. We also support Comfort Noise (CN) and DTMF according to the RTCWEB audio requirements. For video we currently support the H.264UC codec used by Skype services, supporting advanced features such as simulcast, scalable video coding and forward error correction. We’re working toward to enabling interoperable video with H.264.

While the preview implementation may still have bugs, we believe it is ready to support typical scenarios, and are eager to get early hands-on feedback from developers in the Windows Insider Program.

If you are familiar with WebRTC 1.0 implementations, and are interested in the evolution of object support within WebRTC 1.0 and ORTC, we recommend the following presentation by Google, Microsoft, and Hookflash from the 2014 IIT RTC Conference: “ORTC API Update.”

App level code flow for 1:1 communications

Now let’s discuss how to move from the ORTC overview diagram to coding for a simple 1:1 audio/video communication scenario. For this specific scenario, you will need two Win10 machines acting as two peers, and a webserver as the signaling channel to exchange information between the peers so they can establish a connection between them.

The steps below are scoped to operations taken by one peer. Both peers need to go through similar steps in order to set up the 1:1 communications. To make better sense out of the code snippets below, we suggest that you refer to our Microsoft Edge Test Drive Demo as a reference.

Our example uses audio-video and RTP/RTCP multiplexing, so you will see only a single ICE and DTLS transport, used to transport RTP and RTCP packets for both audio and video.

Step #1. Create MediaStream object (i.e. Media Capture API) with one audio track and one video track

.gist table { margin-bottom: 0; }

You can learn more about the Media Capture API in our recent post, Announcing media capture functionality in Microsoft Edge.

Step #2. Create the ICE gatherer, and enable local ICE candidates to be signaled to remote peer.

.gist table { margin-bottom: 0; }

To help protect user privacy, we introduced an option that allows a user to control whether local host IP addresses can be exposed by the IceGatherer objects. We will provide an interface option in Microsoft Edge to toggle this setting.

In our Test Drive Demo we have set up a TURN server. The TURN server has a very limited throughput, so we are limiting that only to our demo page.

Step #3. Create the ICE transport for audio and video, and prepare to handle remote ICE candidates on the ICE transport

.gist table { margin-bottom: 0; }

Another option here is to accumulate all the remote ICE candidates into an array remoteCandidates and call iceTr.setRemoteCandidates(remoteCandidates) to add all the remote candidates together.

Step #4. Create the DTLS transport

var dtlsTr = new RTCDtlsTransport(iceTr);

Step #5. Create the sender and receiver objects

.gist table { margin-bottom: 0; }

Step #6. Retrieve the receiver and sender capabilities

.gist table { margin-bottom: 0; }

Step #7. ICE/DTLS parameters and Send/Receive capabilities can be exchanged.

.gist table { margin-bottom: 0; }

Step #8. Get remote params, start the ICE and DTLS transports, and set the audio and video send and receive parameters.

.gist table { margin-bottom: 0; }

Here is a skeleton of the helper functions. You can find more details in our Test Drive Demo.

.gist table { margin-bottom: 0; }

Step #9. Render and play the remote media stream tracks through a video tag

.gist table { margin-bottom: 0; }

This describes the major steps in the code flow. Please refer to our Test Drive Demo for further details, which include more basic steps in order to set up video preview, and error messages, etc.

Once you are familiar with setting up 1:1 calls, it should be relatively straightforward to set up small group calls using a mesh topology, where each peer will have a 1:1 connection with rest of the group. Since parallel forking is not supported in our platform, this should be handled via 1:1 signaling, so that independent IceGatherer and DtlsTransport objects will be used for each connection.

More ORTC implementation details in Microsoft Edge

We are updating our implementation to match recent updates in the latest ORTC specification. The ORTC spec has been relatively stable since the ORTC CG issued the “Call for Implementations,” and we do not expect a huge challenge at the JavaScript API level. Based on this, we believe our implementation is ready for cross-browser interoperability testing at the protocol level.

There are some limitations in our implementation which we would like to call out:

  • We don’t support RTCIceTransportController. Our implementation handles ICE freezing/unfreezing on a per-transport basis, so that ordering all the IceTransports is not necessary. We believe that this approach should interoperate with existing implementations.
  • RtpListener is not yet supported. This means that SSRCs need to be specified in advance within the RtpReceiver.
  • Forking is not yet supported in either IceTransport, IceGatherer, or DtlsTransport. The solution to DtlsTransport forking is still under discussion in the ORTC CG.
  • RTP/RTCP non-mux with DtlsTransport is not yet supported. When using DtlsTransport your application will need to support RTP/RTCP mux.
  • In RTCRtpEncodingParameters, we currently ignore most of the encoding quality controls. However, we do require setting of the ‘active’ and ‘ssrc’ attributes.
  • The icecandidatepairchanged event is not yet supported. You should be able to extract the candidate pair information through the getNominatedCandidatePair method.
  • We don’t currently support any of the DataChannel functionality currently defined in the ORTC spec.

Looking forward

While our implementation is still early, we’re eager to get your feedback as we work to deliver the full feature set in in Microsoft Edge in the coming months. Our goal is to build an implementation that is interoperable across the web today as well as with the real-time communications industry in the long term.

Towards that goal, in the very near future, the Skype team will leverage the ORTC APIs in Microsoft Edge to enable rich, full-fidelity communication and collaboration experiences, starting with voice and video in the Skype and Skype for Business Web clients. The team also is investing in standards-based WebRTC protocol interoperability, to ensure Skype experiences can run across major desktop, mobile and browser platforms. For developers who want to integrate Skype and Skype for Business experiences into the applications you build, the Skype Web SDK will also leverage ORTC and WebRTC APIs so that your applications will benefit from the same cross platform and browser support. For more information, please check out Enabling Seamless Communication Experiences for the Web with Skype, Skype for Business, and Microsoft Edge.

Furthermore, several members of the ORTC CG have been working closely with us as early technology adopters. We plan to work together to contribute to to the evolution of WebRTC technology toward “WebRTC Next Version (NV).” We expect that early ORTC adopters will be able to share their perspectives soon – stay tuned!

– Shijun Sun, Principal Program Manager, Microsoft Edge
– Hao Yan, Principal Program Manager, Skype
– Bernard Aboba, Architect, Skype

Announcing VP9 support coming to Microsoft Edge

Today, we’re excited to announce that WebM/VP9 support is now in development in Microsoft Edge. VP9 is an open source codec that offers efficient compression to stream HD content at lower bitrates, and is well suited to UHD streaming. Initial support for VP9 will be available in Windows Insider Preview builds soon. This is part of our continuing effort to expand codec offerings in Windows. We continue to evaluate other formats and look forward to receiving feedback as we work on implementing them.

VP9 Support

Our initial VP9 implementation in Microsoft Edge will support adaptive streaming using Media Source Extensions, and will be detectable using the MediaSource.isTypeSupported() API. It will be specifically targeted to meet the needs of websites that use VP9 to deliver video in combination with MP4/AAC or other audio codecs already supported by Microsoft. We are working on future support for VP9 for media tags and local playback, as well as considering support for additional audio formats likely to be used with VP9 such as Opus.

VP9 Usage

Our implementation of VP9 will support software decoding and, when supported by the device, hardware decoding. Since decoding video is computationally complex, the best experience with the software decoder will be seen on more powerful desktop and laptop computers.

Given this, VP9 will initially be implemented behind an experimental flag in Microsoft Edge as we continue to work with industry partners on broader support for hardware decoding, and as we evaluate support for additional audio formats. Users can view their current experimental flag settings and change the defaults by navigating to “about:flags” in the browser. A setting will be available that alternatively enables or disables VP9 support.

Other Formats

We are committed to continually adding new formats to Microsoft Edge to ensure our customers have the best streaming experiences. As part of this commitment, this week Microsoft became a founding member of the newly formed Alliance for Open Media. This announcement represents a commitment from Microsoft and other member companies to specify and develop new open technologies and formats that meet the needs of future media applications. VP9 and other video technologies, such as Thor and Daala, will be a starting point for this new media format work.

Beyond this, there are other existing open source audio and video formats we are evaluating, beginning with OGG, Opus, and Vorbis. We will continue to regularly update Microsoft Edge Platform Status page to identify formats that are in development or under consideration for future releases.

A preview implementation of WebM/VP9 will be available in an upcoming Windows 10 Insider Preview release. We’re eager for your feedback so we can further improve our media format support in Microsoft Edge!

– Andy Glass, Principal Program Manager, Silicon, Graphics & Media
– Jerry Smith, Senior Program Manager, Microsoft Edge

Bringing automated testing to Microsoft Edge through WebDriver

Today we’re announcing support for automated testing of Microsoft Edge through the W3C WebDriver standard. To use WebDriver with Microsoft Edge, you need the MicrosoftWebDriver server on a Windows Insiders build of 10240 or newer. Once that’s installed, you can try out WebDriver for yourself!

WebDriver is an emerging standard through which Web developers can write tests to automate Web browsers for site testing. It provides a programmable remote control for developing complex user scenarios and running them in an automated fashion against your website in a browser. WebDriver is used by top web properties like Bing, Azure, SharePoint, Facebook, Google, and others to automate testing their sites within a browser. With this new capability, Microsoft Edge can be run through the same regression testing as other browsers, helping developers identify issues with less effort and making sites just work for our end users.

As we described in “Building a more interoperable Web with Microsoft Edge,” the Web is built on the principle of multiple independent, interoperable implementations of web standards, and true interoperability means that all web content, browsers, and specifications should align to the same well-defined behavior. Keeping that principle in mind, our implementation supports both the W3C WebDriver specification and the JSON Wire Protocol. The Browser Testing and Tools working group has been doing a great job of standardizing automation testing interfaces as a part of the emerging WebDriver specification. For backwards compatibility with existing tests, we have implemented the JSON Wire Protocol, so existing tests should also just work. We’re not done yet, as the WebDriver specification is still in an early stage of standardization, and we still have more features to implement. You can find out more about all of the WebDriver interfaces we have implemented on our Platform Status page here.

Borland contributions to Microsoft Edge

MicroFocus Borland logoMicrosoft has a long history of collaborating with industry technology leaders to bring the best-in-class experiences. In order to truly build an interoperable implementation that our customers can use, we have partnered with the Borland Silk team from Micro Focus, an industry leader in automation testing tools, to help contribute code to the WebDriver implementation in Microsoft Edge. The Borland team is also bringing their expertise in automation testing to help inform the next level of changes we should pursue in the W3C standard. Our thanks to the Borland team for all of their help in making the Web better!

How WebDriver works

To get started using WebDriver, you will need to download a testing framework of your choice along with an appropriate language binding and the MicrosoftWebDriver server. We have submitted updates to the C# and Java Selenium language bindings and are hoping to add support to other languages in the future.

WebDriver is disabled by default for security. In order to enable using WebDriver, you will need to download and install the MicrosoftWebDriver in a location with your test repository. You should be able to use Microsoft Edge’s WebDriver implementation just like you would use any other browser’s implementation.

Diagram of WebDriver and Microsoft Edge

Here’s an example of C# code opening a new browser Window, navigating to http://www.bing.com and searching webdriver.

.gist table { margin-bottom: 0; }

We’re excited to deliver WebDriver in Microsoft Edge and look forward to hearing what you think! Take it for a spin and share your feedback in the comments below or @MSEdgeDev on Twitter.

Clay Martin, Program Manager, Microsoft Edge
John Jansen, Principal Software Engineer, Microsoft Edge
Jatinder Mann, Senior Program Manager Lead, Microsoft Edge
– Mark Conway, Director, Micro Focus

Announcing the latest improvements for the F12 developer tools in Windows 10

The latest Windows 10 Insider Preview updates bring great new improvements to the F12 developer tools, including some of your most requested features from UserVoice and a new experiments initiative that allows you to try out powerful new features before they’re finished.

We are also happy to announce that our F12 developer tools documentation is now part of the Microsoft Edge Dev site and fully available on GitHub. From this point on, the docs will not only be influenced by your feedback, but you’re invited to contribute and help shape our documentation going forward.

Debugger improvements

Improved CSS editing workflow

As a way to improve the usability and ease of styling your web pages, CSS files are exposed via the Debugger’s filer picker, allowing you to search and select any of the CSS files referenced on the HTML markup.

Debugger’s filer picker

Selecting a file allows you to edit it directly on the Debugger’s text editor. New style changes can be applied via the save button or the Ctrl+S keyboard shortcut.

Edit files directly in the Debugger’s text editor

Cookies

The Debugger now also provides detailed information about cookies that have been created by an HTTP header or via JavaScript. The cookies folder inside the Debugger’s file picker displays a list of the frames loaded by the page. Selecting one of these frames opens up table view of its cookies.

The cookies folder inside the Debugger’s file picker, displays a list of the frames loaded by the page
Selecting one of these frames opens up table view of its cookies.

The cookies view allows you see and modify existing cookies as well as create new ones. You can also clear individual cookies, session cookies or all cookies using the toolbar or the table’s context menu.

Creating a new cookie via the cookies menu

Web storage

Session and local storage are exposed on the debugger’s file picker. Selecting them opens their contents on a new tab that allows you to both inspect and edit their contents.

Selecting and editing local storage via the debugger's file picker

Experiments

The F12 developer tools now have a new experiments tab, which lets you test drive powerful new features that are not fully ready for prime time. These features are disabled by default – you can enable them with the toggles under the Experiments tab.

Our first experiment in the latest build allows you to edit any JavaScript file in the debugger source viewer. Enable the “Edit JavaScript” toggle and restart your browser.

"Edit JavaScript" toggle in the Experiments tab
Once the feature is enabled, simply click on the Debugger’s source viewer to place the cursor and start modifying your JavaScript!

Modifying JavaScript in Debugger source viewer

As you make edits, a dirty flag denoted by an asterisk (*), will show up in the document’s tab. This implies the document has changed but has not yet been saved.

Dirty flag denoted by an asterix indicates that the document has changed but has not yet been saved.

Your edits can then be applied to the page by pressing the save button or using the Ctrl+S keyboard shortcut. Once saved you will be able to compare the modified document to the original document using the diff command.

Diff command in the Debugger

Doing so will open up a diff view of the document.

Diff view of the document

All changes to the document will be lost when you navigate away or refresh from the current page. You will not be able to edit when paused in the debugger or when a document has been pretty printed.

Help us shape our docs!

Over the coming months we will be working on adding new sections and content to enhance the documentation experience. If you’d like to see new coverage or have feedback, please reach out and let us know how we can improve our docs.

For those of you who would like to be more actively involved, we encourage you to visit our GitHub repository and contribute to our documentation by either creating new GitHub issues, editing the existing documentation or even proposing new content. We’ll take a look at your suggestion and will work together to incorporate them to the docs. If you are interested, please visit the repository’s readme to better understand how to contribute.

Remember we value your feedback and we want to encourage you to continue providing feature suggestions, bugs, or requesting experiences you believe can make your daily development activities easier and more impactful. Let us know your thoughts about the features announced today by leaving a comment below, reaching out on Twitter, or visiting our User Voice site to voice your opinion on what set of features we should work on next!

Ruben Rios, Program Manager, Microsoft Edge

Announcing the latest improvements for the F12 developer tools in Windows 10

The latest Windows 10 Insider Preview updates bring great new improvements to the F12 developer tools, including some of your most requested features from UserVoice and a new experiments initiative that allows you to try out powerful new features before they’re finished.

We are also happy to announce that our F12 developer tools documentation is now part of the Microsoft Edge Dev site and fully available on GitHub. From this point on, the docs will not only be influenced by your feedback, but you’re invited to contribute and help shape our documentation going forward.

Debugger improvements

Improved CSS editing workflow

As a way to improve the usability and ease of styling your web pages, CSS files are exposed via the Debugger’s file picker, allowing you to search and select any of the CSS files referenced on the HTML markup.

Debugger’s filer picker

Selecting a file allows you to edit it directly on the Debugger’s text editor. New style changes can be applied via the save button or the Ctrl+S keyboard shortcut.

Edit files directly in the Debugger’s text editor

Cookies

The Debugger now also provides detailed information about cookies that have been created by an HTTP header or via JavaScript. The cookies folder inside the Debugger’s file picker displays a list of the frames loaded by the page. Selecting one of these frames opens up table view of its cookies.

The cookies folder inside the Debugger’s file picker, displays a list of the frames loaded by the page
Selecting one of these frames opens up table view of its cookies.

The cookies view allows you see and modify existing cookies as well as create new ones. You can also clear individual cookies, session cookies or all cookies using the toolbar or the table’s context menu.

Creating a new cookie via the cookies menu

Web storage

Session and local storage are exposed on the debugger’s file picker. Selecting them opens their contents on a new tab that allows you to both inspect and edit their contents.

Selecting and editing local storage via the debugger's file picker

Experiments

The F12 developer tools now have a new experiments tab, which lets you test drive powerful new features that are not fully ready for prime time. These features are disabled by default – you can enable them with the toggles under the Experiments tab.

Our first experiment in the latest build allows you to edit any JavaScript file in the debugger source viewer. Enable the “Edit JavaScript” toggle and restart your browser.

"Edit JavaScript" toggle in the Experiments tab
Once the feature is enabled, simply click on the Debugger’s source viewer to place the cursor and start modifying your JavaScript!

Modifying JavaScript in Debugger source viewer

As you make edits, a dirty flag denoted by an asterisk (*), will show up in the document’s tab. This implies the document has changed but has not yet been saved.

Dirty flag denoted by an asterix indicates that the document has changed but has not yet been saved.

Your edits can then be applied to the page by pressing the save button or using the Ctrl+S keyboard shortcut. Once saved you will be able to compare the modified document to the original document using the diff command.

Diff command in the Debugger

Doing so will open up a diff view of the document.

Diff view of the document

All changes to the document will be lost when you navigate away or refresh from the current page. You will not be able to edit when paused in the debugger or when a document has been pretty printed.

Help us shape our docs!

Over the coming months we will be working on adding new sections and content to enhance the documentation experience. If you’d like to see new coverage or have feedback, please reach out and let us know how we can improve our docs.

For those of you who would like to be more actively involved, we encourage you to visit our GitHub repository and contribute to our documentation by either creating new GitHub issues, editing the existing documentation or even proposing new content. We’ll take a look at your suggestion and will work together to incorporate them to the docs. If you are interested, please visit the repository’s readme to better understand how to contribute.

Remember we value your feedback and we want to encourage you to continue providing feature suggestions, bugs, or requesting experiences you believe can make your daily development activities easier and more impactful. Let us know your thoughts about the features announced today by leaving a comment below, reaching out on Twitter, or visiting our User Voice site to voice your opinion on what set of features we should work on next!

Ruben Rios, Program Manager, Microsoft Edge

Introducing srcset, our first step towards fully responsive images in Microsoft Edge

Windows 10 Insider builds recently introduced support for srcset, our first step on the path to fully supporting the picture element for responsive images. In this post we’d like to share more about our initial implementation of srcset in Windows 10 and our strategy for bringing more components of responsive images to future releases.

What is srcset?

srcset is an attribute on the img element that allows you to declare a set of images and their scale factor (pixel density descriptor). This allows browsers to select the best image for the device currently in use. Here’s an example of an image with various versions defined in its srcset:

<img src="fallback.jpg" srcset="lowres.jpg 1x, mediumres.jpg 1.5x, highres.jpg 2x" />

Note: In order to ship our implementation of srcset in time for Windows 10, Microsoft Edge currently does not support the width (w) descriptor – we’re working on adding this in a future update.

In the above example, the browser collects all of the images in srcset and src, determines which image is best for the user’s device, and only downloads that image. This is a vast improvement over how the majority of srcset polyfills work, as most either do this via a data-* attribute (which has the downside of the src still being downloaded via the pre-parser) or by adding the image via script (which results in potential wasted time to first paint). The srcset attribute allows the browser to take advantage of the pre-parser, but download only the image the user needs.

Picturefill and the RICG, bringing responsive images to the masses

Picturefill is a very popular responsive images polyfill that provides developers with every component of what makes up responsive images:

  • original srcset with density descriptors
  • extended srcset allowing width descriptors
  • the sizes attribute
  • the picture element

Picturefill, along with the Responsive Issues Community Group (RICG) is one of the first libraries to accomplish what the Extensible Web Manifesto is asking for. Picturefill came onto the scene to provide a library that efficiently solves a common problem, and took the additional step of helping it get standardized via the W3C and the WHATWG.

Picturefill is a very popular library and is used on a number of the most popular sites on the Web. This allows authors the freedom to use responsive image technologies today in a progressively enhanced manner, and allows the specification editors to make changes based on author feedback.

How does Picturefill fit in with Microsoft Edge’s srcset implementation?

When we implemented srcset, we initially included the currentSrc API, as detailed in the spec. This API allows a web developer to see which image source was actually selected by the browser. If you’re testing out our implementation of srcset and notice that currentSrc is not working, that’s because we had to remove it; upon an initial flight of this feature, we quickly received numerous bugs about sites not rendering images or redirecting to error pages. After reducing the problem, we discovered that it was caused by an assumption in Picturefill that if the browser doesn’t support the picture element, it must need Picturefill to polyfill all of the components that make up responsive images.

We considered several options in order to get an early implementation out to our Windows Insiders in time to get feedback and discover bugs:

  1. Remove srcset completely and only add it back in once we have picture, sizes and srcset implemented
    We rejected this because we believe there is value to releasing our early implementation of srcset, as evidenced by Safari’s similar implementation. Additionally, it felt like taking a step backward rather than forward.
  2. Make currentSrc writable
    This was initially a tantalizing option as this would provide us with compatibility with the Picturefill polyfill and web developers could use the API freely. However, this approach would result in Microsoft Edge going against the spec, and has potential interoperability impacts. A web developer may inadvertently leave code where they were trying to set currentSrc and no browser would update that currentSrc value besides Microsoft Edge. One of our goals is to follow standards while working with today’s code. We didn’t want to add more confusion and non-standard functionality that holds back the interoperable web – no matter how convenient it would be. Additionally, we had to consider the risk of developers taking a dependency on this capability and not being able to change this easily in the future.
  3. Don’t throw the exception when trying to write to currentSrc
    This is similar to option 2, in that it could risk dependencies in “use strict” mode to not throw errors on currentSrc (such as Picturefill would be if we went this route), would go against the ECMA standard, and would require custom logic in our code just for currentSrc.
  4. Remove currentSrc and ship srcset without it.
    We knew this would be web-compatible as this is how Safari has been shipping srcset. Furthermore, we could keep srcset in the codebase.

Considering all of the above, we decided on option 4, and chose to reach out to the Picturefill team to let them know about the issue with the hopes that it could be patched in a future release of the polyfill. Recently, Safari began hitting this issue and decided to go the same route as Microsoft, further validating this approach.

I want the currentSrc API in Microsoft Edge–what can I do?

Update Picturefill!

In order for us to comfortably release a build to our Insiders with currentSrc enabled, we have to see the usage of Picturefill versions less than 2.3.1 drop tremendously. If you manage a site (or know someone that does) that uses Picturefill, please update to any version later than 2.3.1. The Picturefill team was very prompt in handling our concerns regarding this issue and had a patch in a matter of days put into build 2.3.1.

This is a low-risk update; Mat Marquis of Bocoup (and one of the main contributors of Picturefill and RICG) wrote in a recent CSS Tricks article about this issue: “There haven’t been any breaking changes in any version of Picturefill 2, so you shouldn’t run into any issues updating.”

Additionally, we encourage you to vote for the following features on our User Voice page, as this helps us when ranking new features to implement:

Ultimately, currentSrc will make its way back into Microsoft Edge–it’s just a matter of when. Depending on Picturefill 2.3.1 adoption, we’ll evaluate a preview release to Insiders; otherwise, we’ll wait to restore currentSrc until after we get picture fully implemented.

Do component based feature detection

Responsive images are made up of many different pieces and support for one does not imply the support for another. If you plan to roll your own feature detection, make sure not to do feature detection on just one piece and make an assumption based on that.

Here is an example of how to do feature detection for the various aspects of responsive images:

.gist table { margin-bottom: 0; }

Looking forward

We’re thrilled to preview our early implementation of srcset in Microsoft Edge, as it’s a great start towards lighting up all of the features encompassed in responsive images. I look forward to when we add srcset with width descriptors, sizes and picture in the near future!

Greg Whitworth, Program Manager, Microsoft Edge

Announcing Dolby Audio for high performance audio in Microsoft Edge

Web browser media experiences have advanced significantly in recent years, from the first releases of HTML5 media, through the subsequent additions of Media Source Extensions for adaptive streaming and Encrypted Media Extensions for content protection.  And formats have been similarly advanced by the development of MPEG DASH and Common Encryption (CENC).  These important specifications offered the potential for interoperable, plug-in free web video.  We referred to the resulting experience as Professional Quality Video when we first shipped support for these specs in Internet Explorer 11 on Windows 8.1.

Advancement in media experiences continues in Windows 10 and Microsoft Edge. Today, we are announcing that Microsoft Edge includes HTML5, MSE and EME support for a new audio format:  Dolby Audio™.  It allows websites to match the compelling visuals of H.264 video with equally compelling multi-channel audio.  It works well with AVC/H.264 video and also with our previously announced HLS and MPEG DASH Type 1 streaming features, which both support integrated playback of an HLS or DASH manifest.

Dolby Audio logo

High Performance Media Experiences

We added Dolby Audio to Microsoft Edge so that web apps and sites can deliver the richest and most compelling media experiences with Windows 10.  In particular, we wanted Microsoft Edge to support the most demanding requirements from media websites.  Supporting Dolby Audio fits with this goal.

Dolby Audio

Dolby Audio supports the multi-channel Dolby Digital Plus™ audio format that provides crisp, clear audio in a broad range of channel configurations.  Microsoft Edge is the first browser to include support for Dolby Digital Plus.  Websites will now be able to selectively support Dolby Audio as a premium sound experience, and use HTML5, MSE and EME capability checks to fall back to other supported formats on browsers other than Microsoft Edge.

Windows 10 includes support for the Dolby Digital Plus Portable Mode, which is a dynamic range control setting that is optimized for playback over built-in speakers and headphones. Dolby content will play back louder and clearer with a dynamic range more appropriate for portable devices. This improves the experience from Windows 8 where content could occasionally render softly over built-in speakers and headphones, and the advantages will apply equally to websites running in Microsoft Edge.

Dolby Audio is compatible with current DASH (MP4) and HLS (M2TS) file formats, and so can be included with progressive download HTML5 content, adaptive streaming HLS or MSE/EME MP4 content.

Usage Guidance

There are two adaptive streaming approaches for websites to take advantage of these new formats:

  1. Use either DASH Type 1 streaming (where the DASH manifest (.mpd) or HLS manifest (.m3u8) is set as the src for an HTML 5 media element). Using our DASH Type 1 implementation, it’s important to use the DASH live profile with MPD Type=”static”.
  2. Build a custom adaptive streaming solution using MSE APIs directly. This is more work, but provides the most control over streaming details.

Both the DASH Type 1 and full custom MSE approaches may be combined with Encrypted Media Extensions apply DRM protection.  In Microsoft Edge, this would require supporting PlayReady DRM.

Capabilities

Dolby Audio offers superior performance, but isn’t currently supported by other browsers.  Websites that use it should feature detect on the format and be prepared to stream alternative audio formats on systems that don’t support Dolby Audio.  Javascript examples to check for format support are listed below.

For HTML5:

  • Dolby EC-3: test = myvideo.canPlayType(‘audio/mp4; codecs=”ec-3″‘);
  • Dolby AC-3: test = myvideo.canPlayType(‘audio/mp4; codecs=”ac-3″‘);
  • 264(AVC1): test = myvideo.canPlayType(‘video/mp4; codecs=”avc1.42E01E”‘);
  • 264(AVC3): test = myvideo.canPlayType(‘video/mp4; codecs=”avc3”’);

For MSE:

  • Dolby EC-3: test = MediaSource.isTypeSupported (‘audio/mp4; codecs=”ec-3″‘);
  • Dolby AC-3: test = MediaSource.isTypeSupported (‘audio/mp4; codecs=”ac-3″‘);
  • 264(AVC1): test = MediaSource.isTypeSupported (‘video/mp4; codecs=”avc1.42E01E”‘);
  • 264(AVC3): test = MediaSource.isTypeSupported (‘video/mp4; codecs=”avc3″‘);

For EME:

  • Dolby EC-3: test = MSMediaKeys.isTypeSupported (‘com.microsoft.playready’, ‘audio/mp4; codecs=”ec-3″‘);
  • Dolby AC-3: test = MSMediaKeys.isTypeSupported (‘com.microsoft.playready’, ‘audio/mp4; codecs=”ac-3″‘);
  • 264(AVC1): test = MSMediaKeys.isTypeSupported (‘com.microsoft.playready’, ‘video/mp4; codecs=”avc1.42E01E”‘);
  • 264(AVC3): test = MSMediaKeys.isTypeSupported (‘com.microsoft.playready’, ‘video/mp4; codecs=”avc3″‘);

Dolby Audio Experience Demo

Dolby has created a Dolby Audio Experience demo using WebGL to provide an interactive environment where users can explore and play content on a variety of media devices (a media PC, desktop, laptop and tablet).   Each device demonstrates Dolby Audio with a different streaming video, either MPEG DASH H.264 or HLS.  The demo currently requires Microsoft Edge, since it is the first browser to include native support for Dolby Audio.

Conclusion

The addition of Dolby Audio to Microsoft Edge is intended to open new opportunities for website media experiences.  It is supported only in Microsoft Edge now, but fallback strategies are possible that allow sites to provide the highest quality media experience when supported.

We look forward to your feedback, and hope to see websites using Dolby Audio soon!

– Jerry Smith, Senior Program Manager, Microsoft Edge

Intel’s contributions in Microsoft Edge: SIMD and other performance enhancements

Microsoft has a long history of collaborating with industry technology leaders to bring the world the best-in-class PC and device experiences.  These efforts  now also extend to enhancements in the web browsing experience as well.  Intel is a great example of a key technology partner who is investing and collaborating with Microsoft to advance the web.

Intel has been contributing to Chakra, the JavaScript engine for Microsoft Edge (and previously Internet Explorer), since 2012, bringing their expertise in web runtime development and JIT code generation. Recently, Intel expanded its efforts by contributing to the larger Microsoft Edge codebase, specifically focused in the areas of graphics and performance optimizations. Intel has been a major contributor to open source browser engines such as WebKit, Blink, and Gecko, and with our expanded collaboration, they are now directly contributing to the Microsoft Edge codebase to deliver an improved browsing experience for Windows 10.

Intel’s contributions in the latest Windows 10 Insider Preview

Over the last few years, Intel has been helping with the analysis of specific performance bottlenecks in the Chakra engine. Intel’s direct contributions to Chakra first landed back in the Windows 8.1 timeframe and are ongoing for Windows 10. Some examples of Intel’s direct contributions to Chakra’s JIT compiler include better instruction selection and scheduling. They’ve also helped us identify opportunities of missed redundant instruction elimination, and inline overhead reduction. All up, these changes have helped Chakra reduce code size and perform better for specific payloads.

As a part of broader collaboration between Intel and Microsoft on Windows 10,  Intel engineers are collaborating with us closely to implement Single Instruction Multiple Data (SIMD), a future ECMAScript proposal, in Chakra and Microsoft Edge. SIMD exploits data level parallelism by utilizing specific SIMD hardware instructions, which are supported in most of the modern chipsets regardless of the architecture types. The SIMD instruction set enables performing the same operation on multiple values simultaneously, thus providing much faster code execution when working on data vectors. For example, two data vectors of size four could be added via a single SIMD instruction as compared to four individual add operations. Apart from the performance benefits, getting more done with less instructions also has a positive impact on battery life.

Single Instruction Single Data: Four add operations to add the values in two vectors single instruction multiple data
Single Instruction Single Data:
Four add operations to add the values in two vectors
Single Instruction Multiple Data:
Single add operation to add values in two vectors

SIMD is extremely useful in scenarios like video processing, multimedia, gaming and others. Combined with technologies like asm.js, SIMD could turbocharge and help JavaScript content run at near native speeds. We recently announced SIMD as “In Development” at Microsoft Edge Platform Status, and as a next step, are now previewing SIMD in Chakra and Microsoft Edge for asm.js scenarios. To enable and experiment with SIMD support in Microsoft Edge, navigate to about:flags in Microsoft Edge and select the “Enable asm.js” flag as below:

Screenshot of about:flags interface to enable asm.js

Once the flag is toggled, SIMD will be enabled for code that uses asm.js. While the SIMD JavaScript APIs are available and can be used outside of asm.js scenarios, the performance benefits of SIMD currently accrue and are limited to scenarios that use asm.js running on x86 and x64 hardware. As the SIMD ECMAScript proposal advances and stabilizes, we plan to expand SIMD support in Chakra and Microsoft Edge across all scenarios and hardware.

To see the demo exposing the benefits that SIMD can provide:

  • Navigate to about:flags in Microsoft Edge
  • Toggle the “Enable asm.js” flag
  • Visit the SIMD Mandelbrot demo
  • Use Start, Stop, Use SIMD buttons to run the demo
SIMD Mandelbrot demo with SIMD disabled SIMD Mandelbrot demo with SIMD enabled

In addition to the partnership on Chakra, Intel has recently started contributing to performance improvements in the graphics, layout, and other subsystems of Microsoft Edge as well. As an example, Intel recently contributed an optimization to improve navigation (load) time for pages containing several inline elements, optimizations to reduce DOM parse times for text-area elements, and participated in investigations and root cause analysis to improve page load/response times for the Microsoft Edge browser on Windows 10. And this is only the beginning.

Congratulations to the Intel Team on their contributions to Microsoft Edge so far! We are looking forward to continue to work closely together to move the Web platform forward! Let us know what you think via @MSEdgeDev or in the comments below.

– Rob Hwacinski, Principal Program Manager Lead, Microsoft Edge
– Gaurav Seth, Principal Program Manager Lead, Chakra
– Juan Rodriguez, Senior Principal Engineer, Windows Operating Systems Division, Intel Corporation

Delivering fast JavaScript performance in Microsoft Edge

In Windows 10 and Microsoft Edge we’ve made a lot of performance advances in the Chakra JavaScript engine. All with the goal of letting your JavaScript code run faster, taking as much advantage of the underlying hardware as possible, so that the web you create and experience in Microsoft Edge provides consistently delightful user experiences across all form factors. Amongst others, one of the key themes for our team during this release has been to closely look at data from customer feedback and telemetry reports, and tune the engine to run fast on the existing web, i.e. the web as it exists today.

A while back, we detailed some of the performance enhancements that we’ve made in the Chakra JavaScript engine in Windows 10 for Microsoft Edge. Improving performance is a never ending theme. In this post we detail some enhancements that we’ve done to Chakra’s JIT compiler, based on feedback and telemetry data from the web as it exists today.

Cross file script inlining

All modern JavaScript compilers support function inlining as a key optimization. Function inlining occurs when the function body of a called function is inserted, or inlined, into the caller’s body as if it was part of the caller’s source, thereby saving the overhead of function invocation and return (register saving and restore). In performance sensitive code paths, inlining could provide up to 20-30% performance boost.

While inlining, all compilers need to play a balancing act. For example, beyond a particular size threshold, the compiler might end up spending more time providing the inlined code’s contextual information to produce optimized JIT code than the time it actually saves by the reduced call overhead. Similarly, when trying to inline functions that are coming from another context or script file, the cost of work that needs to be done (like providing the contextual information) could outweigh the performance benefits that inlining provides.

During the development of Windows 10 and Microsoft Edge, we gathered some data from the existing web to understand how Chakra’s inlining optimization was working on the web. We looked at a randomly selected sample of 3,000 sites from the list of top 10,000 sites and following is what we found:

 Diagram showing inlining on top 3000+ sites  Diagram showing per-site breakdown of cross-script inline restrictions
Chakra inlined only 31% of function calls; 48% of the functions were not getting inlined as the caller and callee functions lived in separate script files Using another view, more than half of the sites in our sample were not inlining 60% or more functions

In Windows 10 and Microsoft Edge, Chakra’s JIT compiler and the execution pipeline has been optimized such that Chakra can now efficiently inline functions that are defined across JavaScript files, without losing the performance benefits achieved via inlining. This optimization enables a lot more JavaScript code on the existing web to get inlined and run faster in Microsoft Edge.

Speeding up global constants using fixed field optimization

ECMAScript6 brought the concept of const values to the JavaScript language. Apart from the language and tooling benefits that constants bring to JavaScript developers, they allow JavaScript compilers to optimize look-up performance. When a property is declared as a const, compilers are sure that the value of that property would not change over the lifetime of the program. And given this guarantee, compilers can optimize to avoid look-up costs for such properties. Look up cost includes the cost to check the type/shape/internal representation of the property, figuring out the actual value stored in the property, and checking whether the value has changed during the course of program execution. For constants, the compilers don’t need to do any of the abovementioned checks.

While the usage of const is catching up on the web, a majority of the existing web does not use the const construct today. Instead, in today’s web, most of the times constants are defined once as a variable on the global object and then used everywhere in the code. In one of the experiments we did with 10,000+ sites on the web, we found that more than 20% of the sites had occurrence of such patterns for defining integer constants, and the average was more than 4 such occurrences per site.

In Windows 10 and Microsoft Edge, we’ve started optimizing Chakra’s parser and the JIT compiler to identify non const variable declarations of integers that are defined globally and are never changed during the course of the execution time of the program. Once identified, the JIT’ed code produced by Chakra is able to substantially reduce the lookup cost associated with such globally defined variables that do not change their shape and value during the execution lifetime of the program, thus extending the performance oriented value proposition of the const statement in ECMAScript 6 to how constants are often used in the web as it exists today.

Improving performance of code within try-catch blocks

Using try-catch is very common on the web as it exists today. However, using try-catch is typically not a recommended best practice, especially when the code path is performance sensitive. Try-catch code is hard to optimize because most operations inside the try block can cause an exception and go to the catch.  This flow is too expensive to model precisely in a JIT compiler.  Different techniques need to be used to model this, which create an additional overhead that needs to be taken care of by the execution machinery.

In a data gathering experiment we did on a sample of top 4500 sites, we noticed that more than 96% of the sites threw JavaScript exceptions caught by script code. In fact, more than 50% of sites threw more than 10 JavaScript exceptions that were handled by script code.

Until Windows 10, Chakra did not optimize code inside of try-catch blocks. In Windows 10 and Microsoft Edge, Chakra’s compiler now has the capability to abstract out the code defined inside of the try-catch blocks and generate optimized JIT code for it. For cases where an exception is not thrown, Chakra now executes such code inside a try block almost on par with regular JIT’ed code (as if the try-catch never existed).

Minified code now brings size and speed benefits

Given the advantages of reducing the size of content to be pulled down to a client (browser), usage of minified JavaScript code is common in today’s web. While investigating a particular performance issue during the Windows10 release, one of the things we found was that some instances of minified code (minified using UglifyJS) were not performing as well as their non-minified equivalent were. In some cases minifiers use code patterns that we thought developers typically don’t use, which is why Chakra had not optimized for the same. So we did a quick experiment to see how common minification is on the web. Out of the top 10,000 sites, we took a random sample of around 4,000 sites and below is what we found:

Minification on Top 4000 sites
 Chart showing 95% of sites had some form of minified code  Chart showing that 77% of these sites had code minified using UglifyJS  Chart showing out 47% of those sites used jQuery minified via UglifyJS
95% of the sites had some form of minified code Out of the 95%, 77% sites had some code that was minified using UglifyJS Out of the 95%, 47% of the sites used jQuery minified via UglifyJS

The experiment confirmed that usage of minified code is extremely popular on the web as it exists and amongst others, UglifyJS is very commonly used in today’s web. So in Windows 10 and Microsoft Edge, we’ve added new fast paths, improved inlining and optimized some heuristics in Chakra’s JIT compiler to ensure that minified code runs as fast, if not faster than the non-minified versions. With these changes, the performance of individual code patterns minified using UglifyJS that we tested, improved between 20-50%.

Array#indexOf optimizations

Usage of arrays is pervasive on the web. Apart from providing polyfills and helper functions, a lot of JavaScript libraries out in the wild try to provide faster implementation of some of the standard Array built-ins that come as a part of the JavaScript language. In an ideal world, all browsers should be fast enough on these built-ins such that libraries could focus more on the polyfill and helper API aspect, rather than having to worry about fixing the performance of built-ins across browser implementations. Stating another way, developers should never feel forced to use a library just to make some of the basic built-ins implemented by all JavaScript engines run faster.

Though we are far away from the ideal world mentioned above, in one of the other of data gathering exercises that we recently did, we tried to measure the most used ECMAScript 5 built-ins on the web as it exists today. The experiment was carried out on a random sample of ~4000 of the top 10,000 sites. The top three hits that we found were: Array#indexOf, Array#map and Array#forEach.

Given the popularity of Array built-ins on the web, in Windows10 and Microsoft Edge, Chakra has optimized how values are retrieved, while the engine traverses a given array. This optimization helps remove the extraneous overhead of visiting the prototype chain and looking up the numeric property corresponding to the index, when holes are encountered in an array. This optimization helps improves the performance of ECMAScript5 Array#indexOf built-in in Chakra and Microsoft Edge by more than 5 times.

So, are we fast yet?

Most of the optimizations that we detailed above are based on data from the web as it exists today, and help the existing web run fast in Microsoft Edge. That said, as much as we’d not like to talk about synthetic benchmarks, we often get asked about how (Chakra in) Microsoft Edge performs on popular JavaScript benchmarks. Here’s a look at the performance enhancements that we’ve delivered so far in Microsoft Edge on two JavaScript benchmarks, vis-à-vis IE11 and other popular browsers.

Chart showing Microsoft Edge leading at Octane 2.0 versus IE11, Chrome Canary, and Firefox Alpha, with a score of 23507 Chart showing Microsoft Edge leading at Jet Stream versus IE11, Chrome Canary, and Firefox Alpha, with a score of 154
All measures collected on 64-bit browsers running 64-bit Windows 10 Technical Preview
System Info: HP Compaq 8100 Elite with Intel(R) Core(TM) i7 CPU 860 @ 2.80GHz (4 cores), 12GB RAM

What do these charts tell? Chakra in Microsoft Edge is way faster than IE11. Looking closely, Chakra has made the following improvements across these benchmarks in Microsoft Edge:

Benchmark Owned By Improvements in Microsoft Edge over IE11
Jet Stream Apple More than 1.6 times faster
Octane 2.0 Google More than 2.25 times faster
Note: In case you are curious why these charts measure the performance of 64-bit browsers instead of 32-bit browsers, the reason is that unlike IE11, Microsoft Edge runs the 64-bit web platform underneath whenever possible. While all modern 64-bit JavaScript engines run a tad bit slower than their 32-bit counterparts, choosing a 64-bit web platform provides several security advantages, some of which are outlined in this blog.

While winning on a benchmark that is not created by us does feel nice, the key is that Microsoft Edge has already come a long way from IE11 in terms of improved JavaScript performance on both, benchmarks and real world web as it exists today.  As mentioned in the beginning, performance is a never-ending pursuit. We will continue pushing the performance boundaries for JavaScript in Microsoft Edge. Please keep your feedback coming, as that helps us improve! You can file a bug at Connect , add feedback on UserVoice, or reach out to us on twitter via @MSEdgeDev.

– Gaurav Seth, Principal PM Lead, Chakra

Bringing Web Audio to Microsoft Edge for interoperable gaming and enthusiast media

In March, we released initial preview support for Web Audio to Windows Insiders.  Today, we are excited to share details of our Web Audio implementation in Microsoft Edge.  Web Audio provides tight time synchronization not possible with HTML5 Audio Elements, and includes audio filters and effects useful for enthusiast media apps.  Additionally, Microsoft Edge includes Web Audio support for live audio streams using Media Capture and getUserMedia, and also works with an updated HTML5 Audio Element capable of gapless looping.

Web Audio is specified in the W3C Web Audio Specification, and continues to evolve under the guidance of the W3C Audio Working Group.  This release adds Microsoft Edge to the list of browsers that support Web Audio today, and establishes the specification as a broadly supported, and substantially interoperable web spec.

Web Audio Capabilities

Web Audio is based on concepts that are key to managing and a playing multiple sound sources together.  These start with the AudioContext and a variety of Web Audio nodes.  The AudioContext defines the audio workspace, and the nodes implement a range of audio processing capabilities.  The variety of nodes available, and the ability to connect them in a custom manner in the AudioContext makes Web Audio highly flexible.

Diagram of a conceptual Audio Context

Diagram of a conceptual Audio Context

SourceBufferNodes are typically used to hold small audio fragments.  These get connected to different processing nodes, and eventually to the AudioDestinationNode, which sends the output stream to the audio stack to play through speakers.

In concept, the AudioContext is very similar to the audio graph implemented in XAudio2 to support gaming audio in Classic Windows Applications.  Both can connect audio sources through gain and mixing nodes to play sounds with accurate time synchronization.

Web Audio goes further, and includes a range of audio effects that are supported through a variety of processing nodes.  These include effects that can pan audio across the sound stage (PannerNode), precisely delay individual audio streams (DelayNode), and recreate sound listening environments, like stadiums or concert halls (ConvolverNode).  There are oscillators to generate random signals (OscillatorNode) and many more.

Microsoft Edge all of the following Web Audio interfaces and node types:

Special Media Sources

There are two other node types that Microsoft Edge implements that apply to specific use cases.  These are:

Media Capture integration with Web Audio

The MediaStreamAudioSourceNode accepts stream input from the Media Capture and Streams API, also known as getUserMedia after one of the primary stream capture interfaces.  A recent Microsoft Edge Dev Blog post announced our support for media capture in Microsoft Edge.  As part of this implementation, streams are connected to Web Audio via the MediaStreamAudioSourceNode.  Web apps can create a stream source in the audio context and include stream captured audio from, for example, the system microphone.  Privacy precautions are considered by the specification and discussed in our previous blog.  The positive is that audio streams can be processed in web audio for gaming, music or RTC uses.

Gapless Looping in Audio Elements

The MediaElementAudioSourceNode similarly allows an Audio Element to be connected through Web Audio as well.  This is useful for playing background music or other long form audio that the app doesn’t want to keep entirely in memory.

We’ve both connected Audio & Video Elements to Web Audio, and also made performance changes that allow audio to loop in both with no audio gap.  This allows samples to be looped continuously.

Web Audio Demo

We’ve published a demo to illustrate some of Web Audio’s capabilities using stream capture with getUserMedia.  The Microphone Streaming & Web Audio Demo allows local audio to be recorded and played.  Audio is passed through Web Audio nodes that visualize the audio signals, and apply effects and filters.

The following gives a short walkthrough of the demo implementation.

Create the AudioContext

Setting up the audio context and the audio graph is done with some basic JavaScript.  Simply create nodes that you need (in this case, source, gain, filter, convolver and analyzer nodes), and connect them from one to the next.

Setting up the audioContext is simple:

var audioContext = new (window.AudioContext || window.webkitAudioContext)();

Connect the Audio Nodes

Additional nodes get created by calling node specific create methods on audioContext:

var micGain = audioContext.createGain();
var sourceMix = audioContext.createGain();
var visualizerInput = audioContext.createGain();
var outputGain = audioContext.createGain();
var dynComp = audioContext.createDynamicsCompressor();

Connect the Streaming Source

The streaming source is just as simple to create:

sourceMic = audioContext.createMediaStreamSource(stream);

Nodes are connected from source to processing nodes to the destination node with simple connect calls:

sourceMic.connect(notchFilter);
notchFilter.connect(micGain);
micGain.connect(sourceMix);
sourceAudio.connect(sourceMix);

Mute and Unmute

The mic and speakers have mute controls to manage situations where audio feedback happens.  They are implemented by toggling the gain on nodes at the stream source and just before the AudioDestinationNode:

var toggleGainState = function(elementId, elementClass, outputElement){
     var ele = document.getElementById(elementId);
     return function(){
          if (outputElement.gain.value === 0) {
               outputElement.gain.value = 1;
               ele.classList.remove(elementClass);
          } else {
               outputElement.gain.value = 0;
               ele.classList.add(elementClass);
          }
     };
};
 
var toggleSpeakerMute = toggleGainState('speakerMute', ‘button--selected', outputGain);
var toggleMicMute = toggleGainState('micMute', ‘button--selected', micGain);

Apply Room Effects

Room effects are applied by loading impulse response files into a convolverNode connected in the stream path:

var effects = {
     none: {
          file: 'sounds/impulse-response/trigroom.wav'
     },
     telephone: {
          file: 'sounds/impulse-response/telephone.wav'
     },
     garage: {
          file: 'sounds/impulse-response/parkinggarage.wav'
     },
     muffler: {
          file: 'sounds/impulse-response/muffler.wav'
     }
};
               
var applyEffect = function() {
     var effectName = document.getElementById('effectmic-controls').value;
     var selectedEffect = effects[effectName];
     var effectFile = selectedEffect.file;

Note that we took a small liberty in using the “trigroom” environment as a surrogate for no environmental effect being applied.  No offense is intended for fans of trigonometry!

Visualize the Audio Signal

Visualizations were implemented by configuring analyzerNodes for time and frequency domain data, and using the results to manipulate canvas based presentations.

var drawTime = function() {
     requestAnimationFrame(drawTime);
     timeAnalyser.getByteTimeDomainData(timeDataArray);
 
var drawFreq = function() {
     requestAnimationFrame(drawFreq);
     freqAnalyser.getByteFrequencyData(freqDataArray);

Record & Play

Recorder features use the recorder.js open source sample written by Matt Diamond, and used previously in other Web Audio based recorder demos.  Live audio in the demo uses the MediaStreamAudioSource, but recorded audio is played using the MediaElementAudioSource.  Gapless looping can be tested by activating the loop control during playback.

The complete code for the demo is available for your evaluation and use on GitHub.

Conclusion

There are many articles and other demos available on the web that illustrate Web Audio’s many capabilities, and provide other examples that can be run on Microsoft Edge.  You can try some out on our Test Drive page at Microsoft Edge Dev:

We’re eager for your feedback so we can further improve our Web Audio implementation, and meanwhile we are looking forward to seeing what you do with these new features!

– Jerry Smith, Senior Program Manager, Microsoft Edge