Tag Archives: testing

Cisco refused to participate in NSS Labs report on SD-WAN

Cisco refused to activate the Viptela software-defined WAN product NSS Labs bought for testing, leaving the research firm with a noticeable hole in its recent comparative report on SD-WAN vendors.

Cisco did not provide a reason for refusing to activate the product NSS Labs had purchased for between $30,000 and $40,000, NSS Labs CEO Vikram Phatak said this week. “There was no reason given other than, effectively, they didn’t want to be tested (for the NSS Labs report).”

Cisco’s action marked the first time a vendor had refused to turn on a product NSS Labs had bought for evaluation, Phatak said. Cisco’s Viptela team had initially told NSS Labs it would support the test, which led the firm to buy the product.

“That’s a first for us, candidly,” Phatak said. “And given Cisco’s ethical rules and so on — rules of conduct — I’m in shock because normally, they’re pretty straightforward to work with.”

Cisco refused to discuss the matter, saying in a statement “We believe our customer traction, standing in the market and the continued productive innovation we’re driving speak for themselves.”

NSS Labs wants a refund

NSS Labs wants Cisco to refund the money spent on Viptela. It is hoping it can get the money back without going to court.

“I hope it doesn’t come to that,” Phatak said. “We haven’t talked to any lawyers. I’m assuming that we’ll be able to have the conversation and get our money back.”

Typically, NSS Labs buys products, and the vendors turn them on like they would for any other customer.

“If someone says they don’t want to be tested, we say, ‘That’s great, but if a product is good enough to be sold to the public, it’s good enough to be tested,'” Phatak said. “We’re going to buy it, and we’ll report to the public.”

That’s a first for us, candidly. And given Cisco’s ethical rules and so on — rules of conduct — I’m in shock.
Vikram PhatakCEO, NSS Labs

NSS Labs noted Cisco’s refusal to activate the Viptela purchase in its SD-WAN Comparative Report, which was the company’s first SD-WAN test. Not having Cisco in the evaluation left out one of the largest SD-WAN vendors and a major tech company.

In the first quarter, London-based IHS Markit listed Cisco as No. 4 in the SD-WAN market, just behind Silver Peak. VMware was first with a 19% share, followed by Aryaka with 18%.

The NSS Labs report, released this month, compared the products of nine vendors, including VMware’s NSX SD-WAN, formerly VeloCloud. VMware is Cisco’s largest competitor.

NSS Labs had also planned to include Silver Peak in the comparison but noted it was unable to obtain the product in time for testing.

Tech companies often cite recommended ratings in NSS Labs reports in marketing materials. In April, Cisco highlighted in a blog post the organization’s “recommended” rating for the Cisco Advanced Malware Protection for Endpoints product.

Based on its recent SD-WAN tests, NSS Labs recommended products from VMware, Talari Networks and Fortinet and listed products from Citrix Systems, FatPipe Networks, Forcepoint and Versa Networks as “verified.” Tech buyers should consider recommended and verified products as candidates for purchase, according to NSS Labs.

The company issued “caution” ratings for Barracuda Networks and Cradlepoint, which means companies should not deploy their products without a comprehensive evaluation, NSS Labs said.

NSS Labs ranks next-gen firewalls, with some surprises

New testing of next-generation firewalls found that products from seven vendors effectively protected enterprises from malicious traffic for a reasonable total cost of ownership — under $10 per Mbps of network traffic.

NSS Labs released its annual evaluation of next-gen firewalls on Tuesday, offering seven of 10 product recommendations for security effectiveness and total cost of ownership (TCO) based on comparative testing of hardware and software that prevents unauthorized access to networks.

“Our data shows that north of 80% of enterprises deploy next-gen firewalls,” said Jason Brvenik, CTO at NSS Labs, who noted that the market is mature and many of these vendors’ technologies are in refresh cycles.

The research analysts reviewed next-gen firewalls from 10 vendors for the comparative group test, including:

  • Barracuda Networks CloudGen Firewall F800.CCE v7.2.0;
  • Check Point 15600 Next Generation Threat Prevention Appliance vR80.20;
  • Cisco Firepower 4120 Security Appliance v6.2.2;
  • Forcepoint NGFW 2105 Appliance v6.3.3 build 19153 (Update Package: 1056);
  • Fortinet FortiGate 500E V5.6.3GA build 7858;
  • Palo Alto Networks PA-5220 PAN-OS 8.1.1;
  • SonicWall NSa 2650 SonicOS Enhanced 6.5.0.10-73n;
  • Sophos XG Firewall 750 SFO v17 MR7;
  • Versa Networks FlexVNF 16.1R1-S6; and
  • WatchGuard M670 v12.0.1.B562953.

The independent testing involved some cooperation from participating vendors and in some cases help from consultants who verified that the next-gen firewall technology was configured properly using default settings for physical and virtual test environments. NSS Labs did not evaluate systems from Huawei or Juniper Networks because it could not “verify the products,” which researchers claimed was necessary to measure their effectiveness.

Despite the maturity of the NGFW market, the vast majority of enterprises don’t customize default configurations, according to Brvenik. Network security teams disable core protections that are noisy to avoid false positives and create access control policies, but otherwise they trust the vendors’ default recommendations.

The expanding functionality in next-gen firewalls underscores the complexity of protecting enterprise networks against modern threats. In addition to detecting and blocking malicious traffic through the use of dynamic packet filtering and user-defined security policies, next-gen firewalls integrate intrusion prevention systems (IPS), application and user awareness controls, threat intelligence to block malware, SSL and SSH inspection and, in some cases, support for cloud services.

Some products offer a single management console to enable network security teams to monitor firewall deployments and policies, including VPN and IPS, across environments. An assessment of manageability was not part of NSS Labs’ evaluation, however. NSS Labs focused on the firewall technology itself.

Worth the investment?

Researchers used individual test reports and comparison data to assess security effectiveness, which ranged from 25.0% to 99.7%, and total cost of ownership per protected Mbps, which ranged from U.S. $2 to U.S. $57, to determine the value of investments. The testing resulted in overall ratings of “recommended” for seven next-gen firewalls, two “caution” limited value ratings (Check Point and Sophos) and one “security recommended” but higher than average cost (Cisco).

The security effectiveness assessment was based on the product’s ability to enforce security policies and block attacks while passing nonmalicious traffic over a testing period that lasted several hours. Researchers factored in exploit block rates, evasion techniques, stability and reliability, and performance under different traffic conditions. The total cost of ownership per protected Mbps was calculated using a three-year TCO based on capital expenditure for the products divided by security effectiveness times network throughput.

Six of the next-gen firewalls scored 90.3% or higher for security effectiveness, and most products cost less than $10 per protected Mbps of network throughput, according to the report. While the majority of the next-gen firewalls received favorable assessments, four failed to detect one or more common evasion techniques, which could cause a product to completely miss a class of attacks.

Lack of resilience

NSS Labs added a new test in 2018 for resiliency against modified exploits and, according to the report, none of the devices exhibited resilience against all attack variants.

“The most surprising thing that we saw in this test was that … our research and our testing showed that a fair number of firewalls did not demonstrate resilience against changes in attacks that are already known,” Brvenik said.

Enterprises deploy next-gen firewalls to protect their networks from the internet, he added, and as part of that they expect that employees who browse the internet should not have to worry about new threats. Technology innovation related to cloud integration and real-time updates is promising, but key enterprise problems remain unsolved such as the ability to defend against attacks delivered in JavaScript.

“I think one of the greatest opportunities in the market is to handle that traffic,” said Brvenik, who noted that some next-gen firewalls performed adequately in terms of toolkit-based protections, but NSS Labs didn’t observe any of them “wholly mitigating JavaScript.”

TCO in 2018 is trending lower than previous years. While there are a number of very affordable next-gen firewalls on the market, vendors that can’t validate the effectiveness of next-gen firewalls with independent testing to show the technology can consistently deliver on top-level protections, should be questioned, according to Brvenik. Affordable products are a great choice only if they achieve what the enterprise is looking for and “live up to the security climate.”

UNH InterOperability Lab expands IPv6 testing amid SDN growth

The University of New Hampshire InterOperability Lab updated its IPv6 testing program to comply with new government requirements specified by the National Institute of Standards and Technology. UNH-IOL, a technology testing facility in Durham, N.H., also added support for SDN protocols in its updated program.

The testing program applies specifically to U.S. government agencies, such as NASA, that procure networking equipment and need independent certification that the products meet regulation, according to Timothy Winters, senior IP manager at UNH-IOL. The new requirements come as IPv6 adoption continues to grow globally, as indicated by Google, which said over 20% of its users now have IPv6 addresses, Winters added.

Agencies and product vendors that are UNH-IOL members send devices that need certification to the lab, where UNH students and staff test the products for a month to ensure they support IPv6 and comply.

UNH-IOL tests a range of products, including routers, switches, phones, printers and security cameras. Increasingly, however, agencies and service providers have requested UNH-IOL’s help with SDN and IoT devices, Winters said.

“We’re encountering more devices we haven’t seen,” he said. “Some of this is because of IoT, where things are actually being networked and put on a network. They’re not sitting on a proprietary link anymore.”

IPv6 testing ramps up

Timothy Winters, UNH-IOL senior IP managerTimothy Winters

As operators and service providers realize IPv4 address space is decreasing, they’ve started moving to IPv6-only networks, Winters said. This transition caused UNH-IOL to update its IPv6 testing program accordingly.

“UNH-IOL is trying to push that support, so people building applications and services — or even routers and switches — can know which things work or don’t work in an IPv6-only network,” he said. These changes look at the requirements for building, installing and updating applications — processes that sometimes sound simple, but can actually be quite complicated, he added.

UNH-IOL also patched security loopholes in the IPv6 testing program and made the overall testing more generic, so governments outside the U.S. and other user groups could adopt it, Winters said.

Equipment suppliers have two years to comply with the new IPv6 testing specification. As a result, UNH-IOL will likely see 200 to 300 devices return to the lab to undergo the updated testing, according to Winters.

“I’m sure there are companies that have made some products legacy or don’t sell them anymore, so those won’t come back in,” Winters said. “But that’s a challenge: We have to get everybody back through the program.”

USGv6 testing program flow chart
This flow chart relays the process vendors undergo for IPv6 testing on their products.

IPv6 complements SDN

For us, the exciting part is getting students involved in learning a technology like this. It gives students the ability to build tools, see devices and test them.
Timothy Winterssenior IP manager, UNH-IOL

Additionally, he said the lab now regularly receives routers without a command-line interface to test. This change comes as more service providers and equipment providers find value in SDN — and discover how IPv6 complements SDN deployments, Winters said.

“For SDN, the ability to address multiple services is helpful when you’re trying to get into networks that are so complex they have to be programmed,” he said. Service providers, for example, can use IPv6, along with disaggregation, network slicing and segment routing. The IPv6 address helps identify to which service any particular packet is going.

Along with the other testing updates, UNH-IOL added support for SDN protocols, such as NETCONF and YANG, as well as specs for IoT capabilities. By doing so, Winters said he hopes the lab will help push IPv6 deployments. And, as another plus, UNH-IOL students tackle “the latest and greatest stuff” in networking.

“For us, the exciting part is getting students involved in learning a technology like this,” he said. “It gives students the ability to build tools, see devices and test them.”

Gaming PC – Approx. £550-ish Budget

Testing the waters here, looking for at least:

– Intel CPU such as i7 4970k.
– 16GB RAM.
– Nvidia GFX card.
– Reputable PSU.
– Full ATX build.

I’m not fussed about water cooling or SSD. Components to be from reputable brands, I’d like to avoid generic products (if that makes sense).

I’d be using it for games such as CS 1.6 (I know, I’m an oldie lol) and try out newer games such as PUBG.

I’d be getting my own keyboard and mouse but if there is a quality monitor, I’ll consider too….

Gaming PC – Approx. £550-ish Budget

Cheap GPU

Looking for a cheap gpu for system testing purposes.

Location: London

______________________________________________________
This message is automatically inserted in all classifieds forum threads.
By replying to this thread you agree to abide by the trading rules detailed here.
Please be advised, all buyers and sellers should satisfy themselves that the other party is genuine by providing the following…

Cheap GPU

Cheap Z68 motherboard

Hi

Need a cheap Z68 motherboard to perform testing on some components. Doesnt have to be anything special.

Location: Cambridge / Sudbury

______________________________________________________
This message is automatically inserted in all classifieds forum threads.
By replying to this thread you agree to abide by the trading rules detailed here.
Please be advised, all buyers and sellers should satisfy…

Cheap Z68 motherboard

For Sale – MacBook Pro 13″ 2015. i5 2.9, 8GB RAM, 500GB HD, 1 year’s JL warranty remaining

I’m testing the waters here ……

Somewhat to my surprise, I’m upgrading to a new MacBook so my “old” one will be available by the beginning of next week.

It was bought new by me from John Lewis in February 2016 so has 13 months of the JL warranty remaining. It’s in excellent condition – I take very good care of all my tech – and has been protected by a shell case since day one. If the buyer wishes, I’ll include the case at no extra cost.

Battery count is currently at 348 because my practice is to charge the MacBook, then use it on a tray on my lap until it needs recharging, etc etc. Nothing has been unduly strained by hard usage: mostly, I visit various forums online, browse a number of other sites, and check emails.

This will come with all original packaging and accessories, all in the same excellent condition as the MacBook itself. I’m not sure whether there’s an Apple brown outer box because the unit was collected from JL, but if there isn’t the MacBook will be securely packaged inside another outer box before being posted. Amazon to the rescue again :).

The asking price is inclusive of RMSD. Collection is also possible, should a local buyer want to save a few £££. I’ll get some pics up tomorrow but in the meantime, if you’ve got any questions just ask.

Price and currency: £825
Delivery: Delivery cost is included within my country
Payment method: BT
Location: Bournemouth
Advertised elsewhere?: Not advertised elsewhere
Prefer goods collected?: I have no preference

______________________________________________________
This message is automatically inserted in all classifieds forum threads.
By replying to this thread you agree to abide by the trading rules detailed here.
Please be advised, all buyers and sellers should satisfy themselves that the other party is genuine by providing the following via private conversation to each other after negotiations are complete and prior to dispatching goods and making payment:

  • Landline telephone number. Make a call to check out the area code and number are correct, too
  • Name and address including postcode
  • Valid e-mail address

DO NOT proceed with a deal until you are completely satisfied with all details being correct. It’s in your best interest to check out these details yourself.

For Sale – MacBook Pro 13″ 2015. i5 2.9, 8GB RAM, 500GB HD, 1 year’s JL warranty remaining

I’m testing the waters here ……

Somewhat to my surprise, I’m upgrading to a new MacBook so my “old” one will be available by the beginning of next week.

It was bought new by me from John Lewis in February 2016 so has 13 months of the JL warranty remaining. It’s in excellent condition – I take very good care of all my tech – and has been protected by a shell case since day one. If the buyer wishes, I’ll include the case at no extra cost.

Battery count is currently at 348 because my practice is to charge the MacBook, then use it on a tray on my lap until it needs recharging, etc etc. Nothing has been unduly strained by hard usage: mostly, I visit various forums online, browse a number of other sites, and check emails.

This will come with all original packaging and accessories, all in the same excellent condition as the MacBook itself. I’m not sure whether there’s an Apple brown outer box because the unit was collected from JL, but if there isn’t the MacBook will be securely packaged inside another outer box before being posted. Amazon to the rescue again :).

The asking price is inclusive of RMSD. Collection is also possible, should a local buyer want to save a few £££. I’ll get some pics up tomorrow but in the meantime, if you’ve got any questions just ask.

Price and currency: £825
Delivery: Delivery cost is included within my country
Payment method: BT
Location: Bournemouth
Advertised elsewhere?: Not advertised elsewhere
Prefer goods collected?: I have no preference

______________________________________________________
This message is automatically inserted in all classifieds forum threads.
By replying to this thread you agree to abide by the trading rules detailed here.
Please be advised, all buyers and sellers should satisfy themselves that the other party is genuine by providing the following via private conversation to each other after negotiations are complete and prior to dispatching goods and making payment:

  • Landline telephone number. Make a call to check out the area code and number are correct, too
  • Name and address including postcode
  • Valid e-mail address

DO NOT proceed with a deal until you are completely satisfied with all details being correct. It’s in your best interest to check out these details yourself.

Scale up your deep learning with Batch AI preview

Imagine reducing your training time for an epoch from 30 minutes to 30 seconds, and testing many different hyper-parameter weights in parallel. Available now, in public preview, Batch AI is a new service that helps you train and test deep learning and other AI or machine learning models with the same scale and flexibility used by Microsoft’s data scientists. Managed clusters of GPUs enable you to design larger networks, run experiments in parallel and at scale to reduce iteration time and make development easier and more productive. Spin up a cluster when you need GPUs, then turn them off when you’re done and stop the bill.

Developing powerful AI involves combining large data sets for training with clusters of GPUs for experimenting with network design and optimization of hyper-parameters. Having access to this capability as a service helps data scientists and AI researchers get results faster and focus on building better models instead of managing infrastructure. This is where Batch AI comes in as part of the Microsoft AI platform.

“Deep learning researchers require increasing computing time to train complex neural networks with big data. Large computing clusters on Microsoft Azure is one of the solutions to resolve our researchers’ pain, and Azure Batch AI will be the key solution to connect on-premises and cloud environments. Preferred Networks is excited to integrate Chainer & ChainerMN with this service.” –Hiroshi Maruyama, Chief Strategy Officer, Preferred Networks, Inc

Secret-to-AI

Joseph Sirosh, Corporate Vice President of the Cloud AI Platform, spoke at the recent Microsoft Ignite conference about delivering Cloud AI for every developer with a comprehensive family of infrastructure for AI in Azure, services for AI, and tools to make AI development easier. Batch AI is part of this infrastructure, enabling easily distributed computing on Azure for parallel training, testing, and scoring. Scale-out to as many GPUs as you want.

There’s a great demo in Joseph’s Ignite talk (25 minutes in) that shows an end-to-end experience of data wrangling, training at scale, and using a trained AI model in Excel. The model was developed initially using a Data Science Virtual Machine in Azure, then scaled out to speed up experimentation, hyper-parameter tuning, and training. Using Batch AI, our data scientists were able to scale from 1 to 148 GPUs for the model, reducing training time per epoch from 30 minutes to 30 seconds. This made a huge difference in productivity when you need to run thousands of epochs. Our data scientists were able to experiment with the network design and hyper-parameter values and see results quickly. A version of the code behind this demo will available as a tutorial to use with Batch AI and Azure ML Machine Learning Services and Workbench.

What is Batch AI

Batch AI provides an API and services specialized for AI workflows. The key concepts are clusters and jobs.

Cluster describes the compute resources you want to use. Batch AI enables:

  • Provisioning clusters of GPUs or CPUs on demand

  • Installing software in a container or with a script

  • Automatic or manual scaling to manage costs

  • Access to low priority virtual machines for learning and experimentation

  • Mounting shared storage volumes for training and output data

A job is the code you want to run — a command line with parameters. Batch AI supports:

  • Using any deep learning framework or machine learning tools

  • Direct configuration of options for popular frameworks

  • Priority-based job queue for sharing a GPU quota or reserved instances

  • Restarting jobs if a virtual machine becomes unavailable

  • SDK, command line, portal and tools integration

Building systems of intelligence

Dr. Yogendra Narayan Pandey, Data Scientist at Halliburton Landmark, used Azure Batch AI and Azure Data Lake to develop predictive deep learning algorithms for static reservoir modeling to reduce the time and risk in oil field exploration compared to traditional simulation. He shared his work at the Landmark Innovation Forum & Expo 2017.

“With the huge amounts of storage and compute power of the Azure cloud, we are entering the age of predictive model-based discovery. Batch AI makes it straightforward for data scientists to use the tools they already know. Without Azure Batch AI and GPUs, it would have taken hours if not days for each model training job to complete.”

Batch AI includes recipes for popular AI frameworks that help you get started quickly without needing to learn the details of working with Azure virtual machines, storage, and networking. The recipes include cluster and job templates to use with the Azure CLI interface, as well as Jupyter Notebooks that demonstrate using the Python API.

End-to-end productivity

The Batch AI team is working to integrate with Microsoft AI tools including the Azure Machine Learning services and Workbench for data wrangling, experiment management, deployment of trained models, and Visual Studio Code Tools for AI.

E2E-AI

Partners around the world are also using Batch AI to help their customers scale-up their training to Azure and its powerful fleet of NVIDIA GPUs.

“We have long needed a service like Azure Batch AI. It is an appealing solution for deep learning engineers to speed up deep neural network training & hyper parameter search. I’m looking forward to creating end-to-end solutions by integrating our deep learning service CSLAYER and Azure Batch AI.”  –Ryo Shimizu, President & CEO of UEI Corporation

Getting started

We invite you to try Batch AI for training your models in parallel and at scale in Azure. We have sample recipes for popular AI frameworks to help you get started. We recommend starting with low priority virtual machines to minimize costs.

With Batch AI, you only pay for the compute and storage used for your training. There’s no additional charge for the cluster management and job scheduling. Using low priority virtual machines with Batch AI is the most cost-effective way to learn and develop until you are ready to leverage GPUs.

The team would like to hear any feedback or suggestions you have. We’re listening on Azure Feedback, Stack Overflow, MSDN, and by email.

Artificial intelligence in software testing has arrived

What if software testing could undergo a metamorphosis, becoming better, faster and less expensive all the while letting testers focus on what they excel at? That rosy future could happen, thanks to a sudden interest in artificial intelligence in software testing.

Enterprise solutions provider Infostretch just announced it will offer artificial intelligence in software testing through a brand new service called Predictive and Prescriptive QA. Infostretch isn’t the only option — San Francisco-based startup Appdiff is also bringing machine learning “bots” online as testers. And dinCloud recently announced “James,” a virtual robot QA.

With continuous delivery, continuous integration and DevOps as the hot topics in every software development conversation today, the pressure on testers has never been more intense. “The thing is your crew cannot keep up with the amount of testing that should happen,” Appdiff CEO Jason Arbon said. “That’s one reason for Appdiff. … People can’t keep up any more.”

What about machine learning?

The solution is artificial intelligence in software testing, or more specifically, an AI subset: machine learning. “Today there are tons and tons of test data and it’s very hard for a single person to get through it all,” said Avery Lyford, chief customer officer at Infostretch. “It’s tons of report management now. Where are the real issues and what are the real problems?” That is where artificial intelligence in software testing can come in and help sort through the noise, Lyford said.

Infostretch is offering the Predictive and Prescriptive QA product as a service. With a heavy focus on data analysis, Lyford said the artificial intelligence in software testing tool can help streamline the testing process by ensuring the right information is in the hands of the testers who can then make better decisions. The new service can also be used in conjunction with the company’s QMetry offering.  

AppDiff is taking a slightly different approach, Arbon said. “We’re going from the end user experience backwards,” he said. “AI bots can do tens of thousands of test cases versus 20 to 100 regression test cases. This plays into today’s DevOps plan to iterate quickly.” Using artificial intelligence in software testing, companies will always know if the UI isn’t working or the UX is struggling, he said.

But these aren’t just any bots. Arbon, who previously worked at Google and has a background in software testing, realized a fundamental truth about applications that make bots effective testers. “Almost every app is the same,” he explained. “It’s the same log-in screen, most search boxes look the same, the profile, the shopping carts, there are a lot of similarities.” With that understanding — and the idea that each bot could be trained as a specialist in a single area like just the search box — Arbon was able to create bots that were better than the average tester. “The little bots are specialists on each area of the app and while they’re not as smart as a human might be, they’re the best search testers on the planet.” Arbon, and his colleagues who come from Google and Microsoft, train their bots to test like they did. “It’s like we’ve created a “Google tester” in a box. This replicates what we would do with your app.”

And, amazingly, there may be a silver lining in this for testers, many of whom fear being automated — or AI’d — out of a job. “The folks we work with don’t get fired,” Arbon said. “They get to hand off work and focus on doing things they’re good at.” Or to put it another way, it’s eliminating the grunt work and allowing testers to do the human, creative things they’re better at, said Paul Merrill, principal software engineer in test and founder of Beaufort Fairmont Automated Testing Services, at the Agile2017 conference in Orlando. Lyford sees it as giving testers back that elusive element of time. “We want people to be able to do complicated edge cases, not the routine stuff. This is to augment testers, not replace them.”