IDG Contributor Network: The next big challenge in building the data-driven economy

The importance of data continues to grow, so much so that The Economist recently declared “the world’s most valuable resource is no longer oil, but data.” The rise in connected devices, from mobile phones to information-gathering sensors, is producing more data than ever with the potential to provide new insights about our economy.

As with any valuable resource, though, production is only part of the equation. Logistics — moving the resource to the right place at the right time — is equally important. After all, a gallon of gas only has value when you can get it to people with cars.

When it comes to data, most industries still lag in logistics. While connected devices are driving up data production, many businesses are still using outdated file transfer software, such as FTP, to move vast amounts of data between locations — the digital equivalent of sending packages cross-country by horse and buggy.

Cloud and cognitive fuel need for data transfer

To realize the full potential of data, the technology used to move data will need to catch up. Unlike older transfer systems, these new technologies must provide fast, predictable rates of transfer regardless of the amount of data, locations involved or competing traffic. This need for faster and more advanced data transfer technologies will become even more important as two major trends take hold: the shift to the cloud and the rise of cognitive business.

Cloud is enabling significant cost savings, but also new opportunities.  Mass storage archives are being created in the cloud every day with clients moving vast amounts of data to cheaper storage.  Additionally, cloud-based analytics is enabling the ability to create temporary or permanent analytic stores to answer problems in a much more real-time basis than ever before.  Cognitive application being built in the cloud are only as smart as the data that is given to them.  Incomplete or late arriving data is at times worse than no data at all.

Industry transformation starting to take hold

A few industries have traditionally led the way in making data transfer a priority. IBM Aspera and other technologies that specialize in high-speed data transfer have largely built their businesses through clients in the media and entertainment industry. For these companies, the fast, secure transfer of massive video files has transformed their industry enabling global collaboration and speeds time to market for new projects.

The media and entertainment industry, though, is just the beginning. Some retailers, for example, are now using new data transfer technology to distribute large video and software files to stores globally to more quickly refresh HD video walls and interactive displays. It’s helping them engage shoppers with the latest campaigns and product content and stand out in an intensely competitive field.

Improving health care, advancing auto safety, and fighting fraud

The health care industry also is starting to take advantage of data transfer. It’s helping researchers more efficiently share findings, building their collective body of knowledge more quickly to uncover trends, treatments and cures. Improved data sharing also has the potential to improve care for individual patients by allowing them to get real-time second opinions from specialists thousands of miles away.

In the auto industry, engineers developing next-generation driver assistance software are making use of advanced data transfer tools to collect valuable sensor data from test facilities around the globe. As more technology reaches production, fast data transfer also will be needed between vehicles and infrastructure to underpin new safety features and eventually enable self-driving cars. A lag in even a second could be the difference between a car stopping at the light and cruising into the intersection.

And in banking and financial services, faster and more secure data transfer will become necessary to keep up with the boom in digital transactions and help companies analyze data more quickly. Instead of investigating fraud after it occurs, fast data transfer could help these companies catch criminals in the act.

Data transfer will become a competitive advantage

Data production will continue to grow across all industries. That part of the equation is certain as mobile devices spread globally and information-gathering-sensors fuel the internet of things. The next challenge for businesses is creating the infrastructure to move data to the right place at the right time.

Those companies that master both the mining and movement of data will separate themselves from the pack. They’ll have a more informed view of the market and gain insights more quickly, keeping them in step with the needs of their customers and ahead of their competition.

This article is published as part of the IDG Contributor Network. Want to Join?

Source: InfoWorld Big Data

IDG Contributor Network: Vertically challenged: the pace of digital transformation across industries

As I’ve outlined in my previous posts, we are in the early stages of a digital enterprise transformation tsunami and data is the new currency. By 2020, IDC predicts 1.7 megabytes of new information will be created for every human being on the planet, every second. That’s more data than five times the print collection of the Library of Congress. Every. Year. Never has there been such a motherlode of data for businesses to use to better serve their users and be more efficient. It is a massive challenge for IT organizations to seize this opportunity and fundamentally transform how they manage data and deliver it as a service to their users. It’s a challenge companies in every industry face, but there are some notable differences in how quickly certain industries have embraced the transformation.

Some industries fall into the category of fast “digital transformers”: take the fashion industry, for example. In an article on technology trends shaping the fashion industry, the World Economic Forum notes it is “one major sector being fundamentally transformed from the inside out by technology,” citing evidence like the fashion capitals of the world New York, Paris, and Milan being usurped in importance by digital platforms like Snapchat, Instagram, Pinterest and Periscope. The very nature of stores and the shoppers who used to occupy stores is changing: now that the same clothing items you used to purchase at a store are available online, what incentive does a consumer have to shop at physical stores? Digital transformation is an essential aspect to the success and future of the business, and it’s something the fashion industry, generally, has been very good at recognizing.

What happens to those who don’t take the leap to transform?  Let’s take J.Crew and its struggle to adapt to a tech-first fashion industry as a recent example. Amidst company turmoil, Mickey Drexler, chairman and chief executive of J.Crew Group went as far as to say “If I could go back 10 years, I might have done some things earlier” when referring to embracing technology. Sales at the company have fallen for the past 10 quarters, and the retail veteran who turned J.Crew into a household name actually recently stepped down as chief executive after a failure to stop the brand’s decline.

Competitors with high-tech, data-driven supply chains can copy styles faster, move them into stores quicker, and outmaneuver them. By not pursuing a digital-first approach to its business, J.Crew fell a step behind all the other fashion companies who chose to take the leap sooner. A failure to innovate and digitally transform means companies like this can’t maximize datanomics to their advantage for business intelligence and acceleration; most data simply goes unutilized, gathering dust rather than adding value.

While J.Crew provides us with an example of a digital transformation that may have come too late, the fashion industry does have plenty of examples of faster digital transformations as well. Industries like finance and health care, however, often prove slower to complete the digital transformation necessary to harness modern datanomics. According to a Harvard Business Review article on the biggest health care challenges, health care leaders see outdated or ineffective IT infrastructure as their major roadblock. Oftentimes for these industries battling slow digital transformation, getting sign off from executives to try something new is the biggest hurdle. Knowing the benefits for transforming digitally, regardless of the industry, can often be the first step in convincing your company it’s the right step to take. For health care, critical data underpins the complex interchange of patient records, research, and medical advances, and the controlling data must be reliable, immediately accessible and secure. With this in mind, health care agencies have begun to turn into their own disruptors by implementing data virtualization solutions to collapse costs and time factors, transforming the way they operate and deliver their services in the process.

As an example, a customer of ours, Access Community Health Network, supports 40 community health centers throughout metropolitan Chicago. It is responsible for organizing all patient and financial information. By undertaking digital transformation and implementing a data-as-a-service solution, they were able to streamline, automate, and create 24/7 high-performance access to critical data with no downtime—critical when lives depend on the services they provide.

The finance industry typically mirrors the pace of health care transformation, with the challenge for large banks and financial institutions coming in many forms: for example, a lack of responsiveness and agility in the marketplace worries IT leadership, or slow restores of large databases threaten devops teams. An Actifio customer, a top 20 global consumer and investment bank with data centers around the world, faced the challenge of staying agile in the marketplace and developing new capabilities powered by new software. By taking the leap to transform digitally and improve datanomics, this bank tested and deployed a new database-as-a-service cloud across its entire infrastructure, improving compliance, recovery times, and most importantly the agility of its devops efforts. In the process, the global firm saved well over $25 million in infrastructure, software licensing and operational costs—in the very first year.

No matter the industry and the reputation of industries to transform fast or slow, there is exponentially more data to be managed in digital enterprises today and making sure your organization is equipped to manage that data as a service is crucial. Data is increasingly the most strategic asset of any enterprise, or even an individual. Without proper management of data, it is impossible for organizations to make the most of it to grow a business and leverage the benefits of data-as-a-service. For businesses, managing the datanomics to drive digital transformation is very black and white: either they thrive, or they die. 

This article is published as part of the IDG Contributor Network. Want to Join?

Source: InfoWorld Big Data

Microsoft Azure ExpressRoute Now Available Across Seven CoreSite Markets

CoreSite Realty Corporation has announced the expanded availability of Microsoft Azure ExpressRoute, which can now be privately accessed from seven of CoreSite’s markets across the country, including Northern Virginia, Chicago, Silicon Valley, Denver, Los Angeles, New York, and Boston.

CoreSite customers can privately connect to Microsoft Azure, Office 365 and Dynamics 365 via the CoreSite Open Cloud Exchange, which provides high-performance, SLA-backed virtual connections and on-demand provisioning. The integration of Azure ExpressRoute and the CoreSite Open Cloud Exchange provides CoreSite customers with the opportunity to establish a fast and reliable private connection into Microsoft Azure, Office 365 and Dynamics 365. With an Azure ExpressRoute connection, customers have a natural extension of their data centers and can build hybrid applications that span on-premises infrastructure and Microsoft Cloud services without compromising privacy or performance.

CoreSite customers can efficiently transfer large data sets for high-performance computing, migrate virtual machines between dev-test environments in Azure to production environments housed in a CoreSite data center and optimize replication for business continuity, disaster recovery, and other high-availability strategies.

“We are excited to announce the expanded availability of Microsoft Azure ExpressRoute connectivity to our customers across seven of our key markets,” said Brian Warren, senior vice president of engineering & products at CoreSite. “We are enabling our customers with the solutions necessary to bring together all of their applications, data, devices, and resources, both on-premise and in the cloud, with predictable, reliable, and secure high-throughput connections.”

Source: CloudStrategyMag

Rackspace Expands Private Cloud Capabilities

Rackspace® has announced the general availability of Rackspace Private Cloud powered by VMware®, which will now be built on VMware Cloud Foundation™. With this, customers can enhance the foundational technology that is enabling their move out of the data center and into the cloud with the newest VMware capabilities. Rackspace Private Cloud powered by VMware built on VMware Cloud Foundation will enable full software defined data center (SDDC) capabilities including compute, storage and networking that span the public and private cloud.

VMware Cloud Foundation accelerates IT’s time-to-market by providing a factory-integrated cloud infrastructure stack that is simple to use and includes a complete set of software-defined services for compute, storage, networking and security. Rackspace Private Cloud powered by VMware helps businesses maximize their VMware deployments by helping build, operate and optimize customers’ physical and virtual infrastructure, freeing IT resources from day-to-day infrastructure management so they can focus on their core business. Rackspace is one of the largest global providers in the VMware Cloud Provider™ Program and has partnered with VMware for more than 10 years delivering valuable solutions for mutual customers.

Built on VMware Cloud Foundation, Rackspace Private Cloud provides mutual customers with enhanced capabilities and management benefits including:

• Standardized Architecture: Rackspace Private Cloud powered by VMware is built on VMware Validated Designs, which are based on best practices, making deployments more predictable and lower risk.
• Continuous Updates and Lifecycle Management: Continuous updates allow for the most up-to-date VMware capabilities through lifecycle management of VMware components, thereby helping to improve users’ security posture.
• Leverage Existing VMware Investments: Users leverage the control, flexibility and choice needed to run VMware as easily as they would in their own data center.IT departments can migrate or extend to the VMware cloud with consistent tooling and skills. Consistent infrastructure architecture can be leveraged across multiple locations without the need to refactor code. Mutual customers maintain value of existing investments made in training, VMware technology and familiar tools by accelerating adoption of software-defined infrastructure.
• Offload Physical and Virtual Infrastructure Operations: Rackspace delivers a hosted model, which eliminates many of the procurement and integration challenges that IT organizations face in their own data centers. Mutual customers also benefit from the ability to scale their solution quickly and as needed without the need for significant upfront CAPEX investments in data centers and hardware.
• Managed by Rackspace, Powered by VMware: With Rackspace Private Cloud powered by VMware, customers have access to Fanatical Support® provided 24x7x365 from more than 150 VMware Certified Professionals (VCPs) to help migrate, architect, secure and operate Rackspace hosted clouds powered by VMware technologies.

“Provisioning hardware quickly is no longer considered a value for customers, it’s expected,” said Peter FitzGibbon, vice president and general manager of VMware at Rackspace. “The enhancement in our VMware private cloud delivery model through VMware Cloud Foundation will provide further value to new and existing Rackspace Private Cloud powered by VMware customers by giving them access to the most streamlined and innovative VMware SDDC capabilities and lifecycle management. We are excited to use VMware Cloud Foundation and look forward to continued innovation on the platform.”

“With a decade of proven success in helping customers meet their business demands, VMware and Rackspace are taking another step together to help mutual customers dramatically shorten the path to hybrid cloud,” said Geoffrey Waters, vice president of Global Cloud Sales at VMware. “VMware Cloud Foundation is the industry’s most advanced cloud infrastructure platform that unlocks the benefits of hybrid cloud by establishing a common, simple operational model across private and public clouds. Together with Rackspace and its renowned Fanatical Support, we will add great value to mutual customers in their digital transformation journey.”

Source: CloudStrategyMag

ByteGrid Chosen By Re-Quest, Inc. For Highly Secure Hosting Solutions

ByteGrid Holdings LLC has announced an agreement with Re-Quest, Inc. to provide highly secured technical expertise supporting Re-Quest’s delivery of Oracle hybrid cloud solutions for their customers.

Re-Quest has been successfully assisting customers around the world since 1991, leveraging their investment in Oracle Technology and Infrastructure assets to gain higher returns on investment, lower total cost of ownership and measurable improvement in their business processes.

“Re-Quest prides itself on the high level of business process and technical expertise we bring to every client engagement,” said Ron Zapar, CEO. “Which is why we chose to partner with ByteGrid, providing our customers with high value services across a complete spectrum of Oracle Hybrid Cloud solutions.”

“We know it’s important for Re-Quest to provide their customers with the technical perspective to implement projects that deliver complete customer satisfaction and partner success,” said Jason Silva, ByteGrid’s CTO. “We’re proud to partner with Re-Quest to ensure they’re successful in bringing that satisfaction by hosting their Oracle Hybrid Cloud technology solutions.”

In addition to this new agreement with Re-Quest, ByteGrid serves some of the world’s largest companies and government agencies, including numerous Fortune 50 companies.

Source: CloudStrategyMag

Rob Kakareka Joins Qligent As Manager Of Business Development

Qligent has announced that Robert “Rob” J. Kakareka has joined the company as its new manager/business development. A broadcast industry veteran with extensive sales experience, Kakareka is tasked with developing U.S. sales, customer relationships and market opportunities.

Kakareka reports directly to John Shoemaker, Qligent’s director of sales. Atlanta-based Kakareka will focus on selling the company’s innovative, Vision cloud-based monitoring and compliance platform to U.S. broadcasters, including major networks and call-letter stations.

Qligent’s Vision platform gathers and analyzes data from high-end probes that monitor distinct points along the distribution signal path, out to the last mile. This data enables broadcasters to ascertain that they are delivering an optimal Quality of Experience (QoE) for their viewers, and pinpoint technical issues they need to address.   

“I’m excited to be promoting the value and benefits of Qligent’s flagship product, Vision, at a time of rapid change in the broadcast industry,” said Kakareka. “Vision is uniquely positioned to support mission-critical broadcast distribution in a cost-efficient SaaS model as the industry expands from traditional over-the-air, cable and satellite channels to new digital, mobile and over-the-top (OTT) outlets.

“Despite this dramatic IP-centric shift, the broadcast industry remains a close-knit community with unique requirements and workflows,” Kakareka continued. “My goal is to show broadcasters that not only is our technology exceptional, but we have their backs as they venture into new and emerging market opportunities — including a true Monitoring as a Service business model that offloads monitoring, analysis and troubleshooting responsibilities to our managed services layer.”

With a career spanning over 20 years, Kakareka is no stranger to the broadcast industry, having held strategic sales and business development positions for many high-profile brands. These prior posts include Avid (Orad) Graphics Systems (from February 2014 to February 2016), Miranda (February 2012 to March 2014), Pixel Power (February 2008 to February 2012) and BarcoNet (February 2001 to February 2002).

In these national sales roles, Kakareka regularly outperformed sales quotas, broadened customer bases, boosted sales revenues, and built strong customer relationships with broadcasters nationwide. He’s also knowledgeable in all aspects of broadcast television operations, including graphics and virtual reality studio workflows, SaaS digital media services, big data-scaled storage, TV/film production and OTT/cloud workflows.

Kakareka has also tackled complex business development challenges, such as developing new business for Comprehensive Technical Group, while creating new business plans for this system integration firm’s existing clients. While working for systems integrator Technical Innovations/Broadcast Solutions Group (from February 2002 to February 2008), he implemented a sales plan for the rollout of ATSC compliant DTV systems, sold and integrated them at hundreds of stations across North America, among other sales achievements. 

 “In his stellar career, Rob has witnessed this industry’s many transitions firsthand, and that experience will be especially valuable as we engage with broadcasters to demonstrate how our unique, groundbreaking cloud software can solve today’s ‘uncontained’ distribution challenges,” said Shoemaker. “Our company has experienced rapid growth in a short time, and we’re confident that Rob’s industry expertise, insight and track record will help us capitalize on this momentum and significantly expand our U.S. customer base.”    

Source: CloudStrategyMag

IDG Contributor Network: Responsible retail: treating customer data with care

Retailers have become so adept at capturing and analyzing consumer data that there is now a real risk that they might alienate customers by revealing just much they know about our lifestyle, habits, and preferences. So if retailers want their big data investments to pay off, they must tread carefully. 

Big data exploitation in retail is no longer restricted to tracking and responding to broad trends; it’s become very personal. Which is great if the result is that customers find exactly what they were looking for; less so if it feels intrusive or invasive.

Analytics technology is now so sophisticated that, by drawing on an individual’s loyalty-card records, payment histories and browsing habits, retail marketing programs can detect an alcohol problem, whether someone has lost their job (because spending drops and premium brands are replaced by “value” purchases), if they’re away on holiday, and much more besides. (A few years ago, Target worked out that a teenage girl was pregnant before she knew herself.) 

This is not to imply that retailers are necessarily doing anything wrong or sinister (customers may well have given consent for this kind of data usage). But it can be unnerving to think that every time we browse online or in a store, that activity is being monitored to build a picture of our entire lives. Just think how often we are pestered with unsolicited promotions related to a product we may have glanced at only once.

Even in Europe, where measures to protect consumer privacy are fairly robust, customers are now being tracked via their mobiles as they enter or pass by stores. Their activity can be registered—even if they don’t have a loyalty card or store app. In the US, meanwhile, regulations are becoming looser rather than more stringent now that safeguards protecting internet search histories are being dismantled. So the scope for overstepping the mark is growing.

Snooping vs. problem-solving

If retailers want to impress and retain customers, rather than undermine their trust, they need to turn their attention to more beneficial ways of applying algorithms and data discovery.

In fashion, retailers are exploring ways of minimizing sales returns—a problem so costly across e-commerce that the likes of Amazon have gone so far as banning customers who do this too often. In the US alone, merchandise returns were valued at $260.5 billion in 2015, roughly 8 percent of total sales, according to the National Retail Federation. Returns are a pain for customers, too: who wants the disappointment and hassle of having to send something back because it’s not quite right? A common cause of apparel returns is over-ordering, because consumers haven’t been confident of getting the right size; this is something the industry is now trying to address with new combinations of technology and new data insight.

Another option is to use customer intelligence to provide a more responsive logistics service. Amazon has patented a shipping model that anticipates what goods certain customers are going to order, so it can have the products waiting in a nearby warehouse for faster delivery. Combine this type of strategy with automated drone deliveries and the customer experience might soar while the cost of logistics (even the need for delivery partners) diminishes.

Greater empathy, better service

To the customer, real service innovation reduces the sense of being spied upon because of the perceived personal benefit. The end justifies the means. Just as, if I go to my regular bar, it suits me that they’ll have my favorite drink ready for me before I’ve even taken a seat because of how well they know me. Though if that happened in a bar I’d never been to before, that would be unsettling. Context—and consent—matter.

If the result of deeper customer insight is something genuinely useful to the consumer, surrendering anonymity and sharing data becomes a lot more palatable. People do appreciate easier access to the items they want, it does make their life easier if they don’t have to parcel up returns, and a timely recommendation can be useful in the right circumstances. So really, retailers just need to be a bit more thoughtful about how they apply their knowledge.

What isn’t in dispute is the strategic value of data. Figures from Gallup Behavioral Economics suggest that organizations that are able to exploit customer behavioral insights outperform their peers by 85 percent in sales growth, and more than 25 percent in gross margin. So keep building those data vaults and adding ever more sophisticated real-time analytics; the rest is down to using the insights to best effect.

This article is published as part of the IDG Contributor Network. Want to Join?

Source: InfoWorld Big Data

13 frameworks for mastering machine learning

H2O, now in its third major revision, provides access to machine learning algorithms by way of common development environments (Python, Java, Scala, R), big data systems (Hadoop, Spark), and data sources (HDFS, S3, SQL, NoSQL). H2O is meant to be used as an end-to-end solution for gathering data, building models, and serving predictions. For instance, models can be exported as Java code, allowing predictions to be served on many platforms and in many environments.

H2O can work as a native Python library, or by way of a Jupyter Notebook, or by way of the R language in R Studio. The platform also includes an open source, web-based environment called Flow, exclusive to H2O, which allows interacting with the dataset during the training process, not just before or after. 

Source: InfoWorld Big Data

EvoSwitch Releases White Paper

EvoSwitch has released a new white paper titled: ‘How to Build a Better Cloud –Planning.’ Aimed at CIOs, CTOs and IT Directors, the white paper provides expert-input to a business-driven planning process for weighing multi-cloud environments and implementing a hybrid cloud strategy.

As a colocation services provider with its data centers located in Amsterdam, the Netherlands, and Manassas (Washington DC area) in the U.S., EvoSwitch serves a considerable amount of clients with hybrid cloud needs. That’s why the colocation company established its cloud marketplace, EvoSwitch OpenCloud, two years ago. Through this marketplace, EvoSwitch customers would be able to quickly and securely interconnect to a large number of other cloud platforms including AWS, Google and Azure.

Partly based on these OpenCloud, hybrid cloud customer experiences as well as cloud management expertise of the author himself, the EvoSwitch white paper released today provides CIOs, CTOs, and IT Directors with business-driven guidance for successfully planning their hybrid cloud strategy. Titled ‘How to Build a Better Cloud –Planning,’ the white paper is written by seasoned data center services and cloud computing professional, Patrick van der Wilt, who serves as the commercial director for EvoSwitch.

EvoSwitch’s new white paper ‘How to Build a Better Cloud –Planning’ counts 30 pages and is available in English. It can be downloaded for free here.

Source: CloudStrategyMag

How to use Apache Kafka messaging in .Net

Apache Kafka is an open source, distributed, scalable, high-performance, publish-subscribe message broker. It is a great choice for building systems capable of processing high volumes of data. In this article we’ll look at how we can create a producer and consumer application for Kafka in C#.

To get started using Kafka, you should download Kafka and ZooKeeper and install them on your system. This DZone article contains step-by-step instructions for setting up Kafka and ZooKeeper on Windows. When you have completed the setup, start ZooKeeper and Kafka and meet me back here.

Apache Kafka architecture

In this section, we will examine the architectural components and related terminology in Kafka. Basically, Kafka consists of the following components:

• Kafka Cluster—a collection of one or more servers known as brokers
• Producer – the component that is used to publish messages
• Consumer – the component that is used to retrieve or consume messages
• ZooKeeper – a centralized coordination service used to maintain configuration information across cluster nodes in a distributed environment

The fundamental unit of data in Kafka is a message. A message in Kafka is represented as a key-value pair. Kafka converts all messages into byte arrays. It should be noted that communications between the producers, consumers, and clusters in Kafka use the TCP protocol. Each server in a Kafka cluster is known as a broker. You can scale Kafka horizontally simply by adding additional brokers to the cluster.

The following diagram illustrates the architectural components in Kafka – a high level view.

Apache FOUNDATION

A topic in Kafka represents a logical collection of messages. You can think of it as a feed or category to which a producer can publish messages. Incidentally, a Kafka broker contains one or more topics that are in turn divided into one or more partitions. A partition is defined as an ordered sequence of messages. Partitions are the key to the ability of Kafka to scale dynamically, as partitions are distributed across multiple brokers.

You can have one or more producers that push messages into a cluster at any given point of time. A producer in Kafka publishes messages into a particular topic, and a consumer subscribes to a topic to receive the messages.

Choosing between Kafka and RabbitMQ

Both Kafka and RabbitMQ are popular open source message brokers that have been in wide use for quite some time. When should you choose Kafka over RabbitMQ? The choice depends on a few factors.

RabbitMQ is a fast message broker written in Erlang. Its rich routing capabilities and ability to offer per message acknowledgments are strong reasons to use it. RabbitMQ also provides a user-friendly web interface that you can use to monitor your RabbitMQ server. Take a look at my article to learn how to work with RabbitMQ in .Net.  

However, when it comes to supporting large deployments, Kafka scales much better than RabbitMQ – all you need to do is add more partitions. It should also be noted that RabbitMQ clusters do not tolerate network partitions. If you plan on clustering RabbitMQ servers, you should instead use federations. You can read more about RabbitMQ clusters and network partitions here.

Kafka also clearly outshines RabbitMQ in performance. A single Kafka instance can handle 100K messages per second, versus closer to 20K messages per second for RabbitMQ. Kafka is also a good choice when you want to transmit messages at low latency to support batch consumers, assuming that the consumers could be either online or offline.

Building the Kafka producer and Kafka consumer

In this section we will examine how we can build a producer and consumer for use with Kafka. To do this, we will build two console applications in Visual Studio – one of them will represent the producer and the other the consumer. And we will need to install a Kafka provider for .Net in both the producer and the consumer application.

Incidentally, there are many providers available, but in this post we will be using kafka-net, a native C# client for Apache Kafka. You can install kafka-net via the NuGet package manager from within Visual Studio. You can follow this link to the kafka-net GitHub repository.

Here is the main method for our Kafka producer:

static void Main(string[] args)
{
string payload ="Welcome to Kafka!";
string topic ="IDGTestTopic";
Message msg = new Message(payload);
Uri uri = new Uri(“http://localhost:9092”);
var options = new KafkaOptions(uri);
var router = new BrokerRouter(options);
var client = new Producer(router);
client.SendMessageAsync(topic, new List<Message> { msg }).Wait();
Console.ReadLine();
}

And here is the code for our Kafka consumer:

static void Main(string[] args)
{
string topic ="IDGTestTopic";
Uri uri = new Uri(“http://localhost:9092”);
var options = new KafkaOptions(uri);
var router = new BrokerRouter(options);
var consumer = new Consumer(new ConsumerOptions(topic, router));
foreach (var message in consumer.Consume())
{
Console.WriteLine(Encoding.UTF8.GetString(message.Value));
}
Console.ReadLine();
}

Note that you should include the Kafka namespaces in both the producer and consumer applications as shown below.

using KafkaNet;
using KafkaNet.Model;
using KafkaNet.Protocol;

Finally, just run the producer (producer first) and then the consumer. And that’s it! You should see the message “Welcome to Kafka!” displayed in the consumer console window.

While we have many messaging systems available to choose from—RabbitMQ, MSMQ, IBM MQ Series, etc.—Kafka is ahead of the pack for dealing with large streams of data that can originate from many publishers. Kafka is often used for IoT applications and log aggregation and other use cases that require low latency and strong message delivery guarantees.

If your application needs a fast and scalable message broker, Kafka is a great choice. Stay tuned for more posts on Kafka in this blog.

Source: InfoWorld Big Data