Tag Archives: Streaming

New Confluent Platform release boosts event streaming quality

Event streaming is a critical component of modern data management and analysis, bringing real-time data to organizations. One of the most popular tools for event streaming is the open source Apache Kafka technology that is at the foundation of the commercial Confluent platform.

The vendor, based in Mountain View, Calif., has enhanced the platform with capabilities that make event streaming more secure and resilient.

The Confluent Platform 5.4 event streaming update became generally available Wednesday and benefits from improvements that first landed in the Apache Kafka 2.4 update that was released on Dec. 18. Beyond what’s available in the Kafka update, Confluent’s new release adds role-based access control (RBAC) security, improved disaster recovery and enhanced schema validation for data quality.

Confluent is on a path to improve the usability and manageability of Kafka, said Maureen Fleming, an IDC analyst.

“The introduction of Confluent Schema Registry simplifies and improves control over schema validation and enforcement,” Fleming said. “This aligns well with efforts enterprises are going through to ensure their data is trustworthy.”

The Confluent Platform update also introduces support for CloudEvents, an open specification for describing event data. Fleming noted that improvements in audit logging and support for the CloudEvents specification provide mechanisms for more sophisticated monitoring and use of security-related anomaly detection algorithms.

Screenshot of Confluent schema validation process
Confluent Platform schema validation process

“The improved logging also supports regulatory compliance requirements of Confluent’s customers,” she said.

Securing event data

RBAC is a critical security mechanism that can be used to ensure that only authorized users get access to a given service. Confluent Platform has had integration with directory-based security policy systems, including Microsoft ActiveDirectory, in the past, noted Addison Huddy, group product manager at Confluent. He said the new RBAC system provides more control than what Confluent previously delivered.

The introduction of Confluent Schema Registry simplifies and improves control over schema validation and enforcement. This aligns well with efforts enterprises are going through to ensure their data is trustworthy.
Maureen FlemingAnalyst, IDC

“What role-based access control does is it allows you to take the groups that you have defined already inside of something like Active Directory, and you tie those to roles that we defined in the system,” Huddy said.

Confluent Platform 5.4 has a component that enables administrators to define roles and then have policies on those roles enforced across the platform, he added.

Schema Registry improves data quality

The Confluent Schema Registry is a centralized location where teams can upload their data schemas. Kafka as a platform generally has more users who read and consume data from an event stream than users who write data, Huddy noted.

“So now if I’m writing an application, that’s a consumer of data I don’t have to coordinate with directly to say, “Hey, what serialization format did you use?'” Huddy said. “I can go out and reach the schema registry to grab that data.”

Going a step further, the schema registry can also be used to help enforce data quality, by only accepting data that adheres to a given schema model that is defined in the registry.

KsqlDB now in preview

The new Confluent Platform release also includes a technical preview of the ksqldb event streaming database technology that first became generally available as an open source project on Nov. 20.

One of Confluent customers’ main goals is to build enterprise event streaming applications, said Praveen Rangnath, senior director of product marketing at Confluent. Without ksqlDB, building enterprise event streaming applications would be a more complicated process involving putting multiple distributed systems together, he said.

“What we’re trying to do with ksqlDB is essentially integrate those systems into a single solution to just make it super easy for developers to build event streaming applications,” Rangnath said.

Go to Original Article
Author:

The evolution of music: how the cloud helps reward artists and record labels

From vinyl records to cassette tapes, CDs, MP3 players and streaming services, the way we listen to music has rapidly evolved over the years.

As more and more people turn to the conveniences of streaming music, it’s easy to forget the challenges faced when making sure that artists and labels are fairly compensated.

Everyone talks about digital transformation providing companies with a competitive edge. For music rights organisations, however, adopting a digital culture isn’t a choice. It’s a matter of survival.

This was the situation facing the Canadian Musical Reproduction Rights Agency (CMRRA), a music licensing organisation which represents the majority of music publishers and music copyright owners in Canada.

In 2011, the CMRRA along with the rest of the music copyright industry faced drastic changes to its business model. Streaming services such as Spotify and Apple Music were dramatically increasing the number of transactions from tens of thousands to hundreds of millions, while the revenue per transaction decreased to small fractions of a cent.

In this new digital-first world, a file containing hundreds of millions of transactions can generate royalty payments of €100,000. In the pre-digital world, for comparison, this would have generated millions of Euros instead.

To help its transformation, CMRRA needed a robust and secure solution which would help with the increased number of transactions in a cost-effective way, while allowing it to continue to distribute royalties to artists and other rights holders. The company began its journey by turning to Spanish Point – a Microsoft Gold Partner in Ireland.

Hitting play on transformation
Spanish Point had already digitised the process for the Irish Music Rights Organisation (IMRO), but it wasn’t just a case of dealing with increased transaction volumes, as Spanish Point CEO Donal Cullen explains: “There is also the problem of matching millions of music streaming transactions with poor metadata against a database of millions of songs,” he states.

“Many copyright organisations have failed to cope with this increase in data volumes, meaning the songs and recordings have not been licensed or correctly identified. The license income that should have been paid to songwriters and music publishers has remained with streaming companies. The streaming services and other entertainment platforms do want to pay the artists, it’s just a question of finding a practical way of doing it.”

The solution developed by Spanish Point saw CMRRA move its operations to the cloud, enabling it to successfully cope with these challenges and generate more income for its members. Using Microsoft’s advanced features, Spanish Point provided a more agile and responsive service at a lower cost than a traditional on-premise or hosted provider.

“In the past, if a song was played on a radio station it was broadcast to thousands of listeners,” Cullen explains. “Now you have people using their smartphone in cars to stream music. Services like Spotify and YouTube are sending data to rights organisations on each individual stream. That has increased the volume of data by three or four orders of magnitude.”

“It is not unusual for files to contain 200 million transactions. The rights organisations now must identify each song from quite poor metadata and find the artists to pay royalties to. There is simply no way they could do that without a cloud solution. Even four or five years ago it would have been beyond our reach. It has enabled us to help customers like CMRRA improve their data processing performance by a factor of 40.”

Moving to the cloud solved the problems of scale, flexibility and financial viability. “Before the cloud, organisations would invest in computing power to meet peak demand,” Cullen notes. “That meant the payroll system had to be able to meet very high demand on one or two days each month while it would be barely used for the rest of the time. In the cloud you pay for what you use as you need it. Also, Microsoft’s cloud autoscales to meet the size of files and that’s directly related to how much we and our customers are going to get paid.”

Microsoft Ireland commercial director Aisling Curtis believes the challenge faced by the music rights industry demonstrates the enormous power of digital culture. “This is a great example of digital disruption and how a digital transformation approach can be used to solve issues across an entire industry”, she says.

“It’s not just something for large companies or enterprise-sized organisations to be concerned about. Organisations of every size can adopt a digital culture to innovate and gain competitive advantage. Spanish Point has done a fantastic job for CMRRA using the Microsoft platform and has created a new solution which is applicable to the whole music rights industry.”

Cloud with benefits
As a result of its transformation, CMRRA has dramatically increased its revenue and reduced its members’ annual subscription fees from 10.5 per cent to six per cent. It has also opened up new numerous new opportunities for the company.

“They are now going to licence mechanical works in the US,” says Cullen. “They were restricted to Canada up until now, but they have become a lot cheaper than their US competitors because of the Microsoft cloud solution.”

For the future, Spanish Point is planning to use Microsoft’s AI technology to further enhance its solution. Currently, the company is moving into the US market and is also working with customers in Spain and Turkey with this solution.

“Microsoft has worked closely with Spanish Point on a number of digital transformation projects over the years,” says Aisling Curtis. “Spanish Point is a very innovative firm. It explores new frontiers with Microsoft products and platforms which enables its customers to access new business opportunities and gain competitive advantage. This is a very tangible example of how digital culture and transformation is allowing an Irish company to solve a worldwide issue for customers. It is a defining example of the impact of digital culture.”

Confluent Platform 5.0 aims to mainstream Kafka streaming

The Confluent Platform continues to expand on capabilities useful for Kafka-based data streaming, with additions that are part of a 5.0 release now available from Confluent Inc.

The creation of former LinkedIn data engineers who helped build the Kafka messaging framework, Confluent Platform’s goal is to make real-time big data analytics accessible to a wider community.

Part of that effort takes the form of KSQL, which is meant to bring easier SQL-style queries to analytics on Kafka data. KSQL is a Kafka-savvy SQL query engine and language Confluent created in 2017 to open Kafka streaming data to analytics.

Version 5.0 of the Confluent Platform, commercially released on July 31, seeks to improve disaster recovery with more adept handling of application client failover to enhance IoT abilities with MQTT proxy support, and to reduce the need to use Java for programming streaming analytics with a new GUI for writing KSQL code.

Data dips into mainstream

Confluent Platform 5.0’s support for disaster recovery and other improvements is useful, said Doug Henschen, a principal analyst at Constellation Research. But the bigger value in the release, he said, is in KSQL’s potential for “the mainstreaming of streaming analytics.”

Doug Henschen, Constellation ResearchDoug Henschen

Besides the new GUI, this Confluent release upgrades the KSQL engine with support for user-defined functions, which are essential parts of many existing SQL workloads. Also, the release supports handling nested data in popular Avro and JSON formats.

“With these moves Confluent is meeting developer expectations and delivering sought-after capabilities in the context of next-generation streaming applications,” Henschen said.

That’s important because web, cloud and IoT applications are creating data at a prodigious rate, and companies are looking to analyze that data as part of real-time operations. The programming skills required to do that level of development remain rare, but, as big data ecosystem software like Apache Spark and Kafka find wider use, simpler libraries and interfaces are appearing to link data streaming and analytics more easily.

Kafka, take a log

At its base, Kafka is a log-oriented publish-and-subscribe messaging system created to handle the data created by burgeoning web and cloud activity at social media giant LinkedIn.

The core software has been open sourced as Apache Kafka. Key Kafka messaging framework originators, including Jay Krebs, Neha Narkhede and others, left LinkedIn in 2014 to found Confluent, with the stated intent to build on core Kafka messaging for further enterprise purposes.

Joanna Schloss, Confluent’s director of product marketing, said Confluent Platform’s support for nested data in Avro and JSON will enable greater use of business intelligence (BI) tools in Kafka data streaming. In addition, KSQL now support more complex joins, allowing KSQL applications to enhance data in more varied ways.

Joanna Schloss, director of product marketing at ConfluentJoanna Schloss

She said opening KSQL activity to view via a GUI makes KSQL a full citizen in modern development teams in which programmers, as well as DevOps and operations staff, all take part in data streaming efforts.

“Among developers, DevOps and operations personnel there are persons interested in seeing how Kafka clusters are performing,” she said. Now, with the KSQL GUI, “when something arrives they can use SQL [skills] to watch what happened.” They don’t need to find a Java developer to interrogate the system, she noted.

Making Kafka more accessible for applications

KSQL is among the streaming analytics capabilities of interest to Stephane Maarek, CEO at DataCumulus, a Paris-based firm focused on Java, Scala and Kafka training and consulting.

Stephane Maarek, CEO of DataCumulusStephane Maarek

Maarek said KSQL has potential to encapsulate a lot of programming complexity, and, in turn, to lower the barrier to writing streaming applications. In this, Maarek said, Confluent is helping make Kafka more accessible “to a variety of use cases and data sources.”

Moreover, because the open source community that supports Kafka “is strong, the real-time applications are really easy to create and operate,” Maarek added.

Advances in the replication capabilities in Confluent Platform are “a leap forward for disaster recovery, which has to date been something of a pain point,” he said.

Maarek also said he welcomed recent updates to Confluent Control Center, because they give developers and administrators more insights into the activity of Kafka cluster components, particularly schema registry and application consumption lags — the difference between messaging reads and messaging writes. The updates also reduce the need for administrators to write commands, according to Maarek.

Data streaming field

The data streaming field remains young, and Confluent faces competition from established data analytics players like IBM, Teradata and SAS Institute, Hadoop distribution vendors like Cloudera, Hortonworks and MapR, and a variety of specialists such as MemSQL, SQLstream and Striim.

“There’s huge interest in streaming applications and near-real-time analytics, but it’s a green space,” Henschen said. “There are lots of ways to do it and lots of vendor camps — database, messaging-streaming platforms, next-gen data platforms and so on — all vying for a piece of the action.”

However, Kafka often is a common ingredient, Henschen noted. Such ubiquity helps put Confluent in a position “to extend the open source core with broader capabilities in a commercial offering,” he said.

For Sale – i7 4770k Bundle (CPU, RAM and Motherboard)

Selling my streaming computer to fund other things. Not really looking to split right now but may do in the future.

CPU: i7 4770k
RAM: Corsair 8GB DDR3 1600Mhz 9-9-9-24
MOBO: ASUS Z87-EXPERT

Unfortunately, I do not have the original boxes for any of these items, but I can verify everything works perfectly.

Price and currency: £200
Delivery: Delivery cost is included within my country
Payment method: PPG or BT
Location: London
Advertised elsewhere?: Advertised elsewhere
Prefer goods collected?: I have no preference

______________________________________________________
This message is automatically inserted in all classifieds forum threads.
By replying to this thread you agree to abide by the trading rules detailed here.
Please be advised, all buyers and sellers should satisfy themselves that the other party is genuine by providing the following via private conversation to each other after negotiations are complete and prior to dispatching goods and making payment:

  • Landline telephone number. Make a call to check out the area code and number are correct, too
  • Name and address including postcode
  • Valid e-mail address

DO NOT proceed with a deal until you are completely satisfied with all details being correct. It’s in your best interest to check out these details yourself.

For Sale – i7 4770k Bundle (CPU, RAM and Motherboard)

Selling my streaming computer to fund other things. Not really looking to split right now but may do in the future.

CPU: i7 4770k
RAM: Corsair 8GB DDR3 1600Mhz 9-9-9-24
MOBO: ASUS Z87-EXPERT

Unfortunately, I do not have the original boxes for any of these items, but I can verify everything works perfectly.

Price and currency: £200
Delivery: Delivery cost is included within my country
Payment method: PPG or BT
Location: London
Advertised elsewhere?: Advertised elsewhere
Prefer goods collected?: I have no preference

______________________________________________________
This message is automatically inserted in all classifieds forum threads.
By replying to this thread you agree to abide by the trading rules detailed here.
Please be advised, all buyers and sellers should satisfy themselves that the other party is genuine by providing the following via private conversation to each other after negotiations are complete and prior to dispatching goods and making payment:

  • Landline telephone number. Make a call to check out the area code and number are correct, too
  • Name and address including postcode
  • Valid e-mail address

DO NOT proceed with a deal until you are completely satisfied with all details being correct. It’s in your best interest to check out these details yourself.

For Sale – i7 4770k Bundle (CPU, RAM and Motherboard)

Selling my streaming computer to fund other things. Not really looking to split right now but may do in the future.

CPU: i7 4770k
RAM: Corsair 8GB DDR3 1600Mhz 9-9-9-24
MOBO: ASUS Z87-EXPERT

Unfortunately, I do not have the original boxes for any of these items, but I can verify everything works perfectly.

Price and currency: £200
Delivery: Delivery cost is included within my country
Payment method: PPG or BT
Location: London
Advertised elsewhere?: Advertised elsewhere
Prefer goods collected?: I have no preference

______________________________________________________
This message is automatically inserted in all classifieds forum threads.
By replying to this thread you agree to abide by the trading rules detailed here.
Please be advised, all buyers and sellers should satisfy themselves that the other party is genuine by providing the following via private conversation to each other after negotiations are complete and prior to dispatching goods and making payment:

  • Landline telephone number. Make a call to check out the area code and number are correct, too
  • Name and address including postcode
  • Valid e-mail address

DO NOT proceed with a deal until you are completely satisfied with all details being correct. It’s in your best interest to check out these details yourself.

For Sale – i7 4770k Bundle (CPU, RAM and Motherboard)

Selling my streaming computer to fund other things. Not really looking to split right now but may do in the future.

CPU: i7 4770k
RAM: Corsair 8GB DDR3 1600Mhz 9-9-9-24
MOBO: ASUS Z87-EXPERT

Unfortunately, I do not have the original boxes for any of these items, but I can verify everything works perfectly.

Price and currency: £200
Delivery: Delivery cost is included within my country
Payment method: PPG or BT
Location: London
Advertised elsewhere?: Advertised elsewhere
Prefer goods collected?: I have no preference

______________________________________________________
This message is automatically inserted in all classifieds forum threads.
By replying to this thread you agree to abide by the trading rules detailed here.
Please be advised, all buyers and sellers should satisfy themselves that the other party is genuine by providing the following via private conversation to each other after negotiations are complete and prior to dispatching goods and making payment:

  • Landline telephone number. Make a call to check out the area code and number are correct, too
  • Name and address including postcode
  • Valid e-mail address

DO NOT proceed with a deal until you are completely satisfied with all details being correct. It’s in your best interest to check out these details yourself.

For Sale – i7 4770k Bundle (CPU, RAM and Motherboard)

Selling my streaming computer to fund other things. Not really looking to split right now but may do in the future.

CPU: i7 4770k
RAM: Corsair 8GB DDR3 1600Mhz 9-9-9-24
MOBO: ASUS Z87-EXPERT

Unfortunately, I do not have the original boxes for any of these items, but I can verify everything works perfectly.

Price and currency: £200
Delivery: Delivery cost is included within my country
Payment method: PPG or BT
Location: London
Advertised elsewhere?: Advertised elsewhere
Prefer goods collected?: I have no preference

______________________________________________________
This message is automatically inserted in all classifieds forum threads.
By replying to this thread you agree to abide by the trading rules detailed here.
Please be advised, all buyers and sellers should satisfy themselves that the other party is genuine by providing the following via private conversation to each other after negotiations are complete and prior to dispatching goods and making payment:

  • Landline telephone number. Make a call to check out the area code and number are correct, too
  • Name and address including postcode
  • Valid e-mail address

DO NOT proceed with a deal until you are completely satisfied with all details being correct. It’s in your best interest to check out these details yourself.

For Sale – i7 4770k Bundle (CPU, RAM and Motherboard)

Selling my streaming computer to fund other things. Not really looking to split right now but may do in the future.

CPU: i7 4770k
RAM: Corsair 8GB DDR3 1600Mhz 9-9-9-24
MOBO: ASUS Z87-EXPERT

Unfortunately, I do not have the original boxes for any of these items, but I can verify everything works perfectly.

Price and currency: £200
Delivery: Delivery cost is included within my country
Payment method: PPG or BT
Location: London
Advertised elsewhere?: Advertised elsewhere
Prefer goods collected?: I have no preference

______________________________________________________
This message is automatically inserted in all classifieds forum threads.
By replying to this thread you agree to abide by the trading rules detailed here.
Please be advised, all buyers and sellers should satisfy themselves that the other party is genuine by providing the following via private conversation to each other after negotiations are complete and prior to dispatching goods and making payment:

  • Landline telephone number. Make a call to check out the area code and number are correct, too
  • Name and address including postcode
  • Valid e-mail address

DO NOT proceed with a deal until you are completely satisfied with all details being correct. It’s in your best interest to check out these details yourself.

For Sale – i7 4770k Bundle (CPU, RAM and Motherboard)

Selling my streaming computer to fund other things. Not really looking to split right now but may do in the future.

CPU: i7 4770k
RAM: Corsair 8GB DDR3 1600Mhz 9-9-9-24
MOBO: ASUS Z87-EXPERT

Unfortunately, I do not have the original boxes for any of these items, but I can verify everything works perfectly.

Price and currency: £200
Delivery: Delivery cost is included within my country
Payment method: PPG or BT
Location: London
Advertised elsewhere?: Advertised elsewhere
Prefer goods collected?: I have no preference

______________________________________________________
This message is automatically inserted in all classifieds forum threads.
By replying to this thread you agree to abide by the trading rules detailed here.
Please be advised, all buyers and sellers should satisfy themselves that the other party is genuine by providing the following via private conversation to each other after negotiations are complete and prior to dispatching goods and making payment:

  • Landline telephone number. Make a call to check out the area code and number are correct, too
  • Name and address including postcode
  • Valid e-mail address

DO NOT proceed with a deal until you are completely satisfied with all details being correct. It’s in your best interest to check out these details yourself.