You've requested...

Designing for the Cloud: What Today's Data Engineer Should be Considering When Building Their Stack

If a new window did not open, click here to view this asset.

Download this next:

Spark Muscles Into Big Data Processing

It comes as no great surprise that the Apache Spark architecture has been horning in on the batch processing domain once controlled by Hadoop's MapReduce. But that's only part of the story. With data processing, streaming and machine learning capabilities on its résumé, the open source engine is learning to get along entirely without Hadoop in certain applications. In fact, one industry analyst cautiously sees a day when Spark could declare total independence, potentially bust up Hadoop cluster dominance and link separately with other Apache technologies. 

In this three-part handbook, senior news writer Jack Vaughan examines the distinct advantages the Apache Spark architecture has over MapReduce. Also highlighted is how Spark's ability to process and analyze streaming data is helping detect fraudulent activities at a major banking and credit-card company. Next, Spark 2.0's upcoming upgrades to analytics speed, machine learning libraries, SQL support and stream processing are detailed. To close, Vaughan and senior news writer Ed Burns look at combining Spark and NoSQL databases in operational analytics applications, which could help broaden the use of both technologies.

These are also closely related to: "Designing for the Cloud: What Today's Data Engineer Should be Considering When Building Their Stack"

  • What data security looks like in 2024

    These days, your sensitive data likely exists in multiple places, both outside of and within your company.

    How can you ensure that your data is safe in all its locations? This overview explains how a data security posture management solution can help by:

    • Boosting visibility into user data activity
    • Enabling IT flexibility
    • And more

    Check out the overview to learn about one such solution: IBM Security Guardium Insights.

  • FAQ: Hadoop for BI and Analytics

    Traditional data warehouses are proving to be ill-suited for tackling big data on their own. As a result, organizations are turning to new technologies like Hadoop to effectively manage today’s data environment.

    Unfortunately, Hadoop is surrounded by many misconceptions and most data management professionals have little to no experience with the big data platform.

    This expert E-Guide aims to dispel common Hadoop myths and offers key advice for leveraging this technology in order to drive big data analytics success. Read on to learn more.

Find more content like what you just read:

  • Rethink data integration for the age of big data

    This expert e-guide explores the need for new thinking around data integration in a big data world, and highlights key tools that can deliver the value you're looking for.

    Download

  • Riding the elephant: how to manage big data

    Big data vendors – such as the Hadoop distributors – will say they see signs that big data science projects are giving way to large scale implementations. But how are user organisations managing those? How are they designing their IT organisations, and the rest, to manage and capitalise on big data?

    Download

  • Top 10 information management stories of 2019

    Consolidation and acquisition marked the supplier side of the IT industry in respect of information management in 2019. The question of how modern BI and analytics software shapes up to nonrelational big data is addressed. Here are Computer Weekly's top 10 information management stories of 2019.

    Download

  • Computer Weekly – 20 June 2017: Why did British Airways' datacentre crash?

    After the recent hastily held general election, the former Cabinet Office minster Ben Gummer lost his Ipswich seat to one of Theresa May's close allies, leaving many wondering if a digital future is still a top priority in the new government. Find the answer to this question, as well as information on the cause of the British Airways IT failure.

    Download

  • Hadoop Data Management Platforms and Appliances

    This expert Technology Guide provides an introduction to Hadoop and suggests criteria for selecting a Hadoop data management platform. Also included are reviews of leading Hadoop systems from a variety of vendors.

    Download

  • A Computer Weekly buyer's guide to Business Intelligence and Analytics

    Business intelligence and analytics is an ever-changing landscape. In this 20-page buyer's guide, Computer Weekly looks at firms' search for data scientists, how modern business intelligence works for big data and the role DevOps can play in navigating a data onslaught.

    Download

  • A Computer Weekly buyer's guide to NoSQL

    There's a lot of data out there, much of which doesn't need to be stored in heavyweight relational databases complete with complex query languages. In this 13-page buyer's guide, Computer Weekly looks at a new category of NoSQL data stores and how they are being used.

    Download

  • Navigating The Twists and Turns of The Hadoop Landscape

    It can be difficult to navigate the big data landscape. It's hard to ignore how rapidly Hadoop-based products are transforming, so many IT shops are wondering what the future of Hadoop has in store. This e-guide provides expert insights into the recent changes and their impact on IT.

    Download

  • Computer Weekly – 5 June 2018: The bumpy ride to digital transformation

    In this week's Computer Weekly, we look at the lessons to be learned from General Electric's bumpy ride to digital transformation. We assess the use of software-defined networking in corporate IT infrastructures. And we find out how Royal Bank of Scotland approaches digital and business innovation. Read the issue now.

    Download

  • Preparing a database strategy for Big Data

    This e-guide outlines the key database challenges and opportunities introduced by big data, and offers expert advice to help you successfully prepare a big data database strategy.

    Download

  • A Computer Weekly Buyer's Guide to Enterprise Open Source

    In this 13-page buyer's guide, Computer Weekly looks at open source technology. Discover more about why open source has become the infrastructure choice over commercial software; the due diligence required to reap the rewards of open source and how to choose the best fit for your needs.

    Download

  • Object storage draws on metadata capability

    In this guide, Ellen O'Brien – the Executive Editor for SearchStorage.com – discusses how object storage is in high-demand for archiving and analyzing big data. Discover how metadata capabilities are propelling object storage beyond cloud storage and into a number of new and exciting use cases.

    Download

  • Who will win the world's biggest cloud contract?

    In this week's Computer Weekly, we examine the $10bn Pentagon cloud contract – codenamed JEDI – and ask why it is taking so long to choose a supplier. We discuss with IT chiefs how AI will transform technology leadership. And we look at how to improve your email security. Read the issue now.

    Download

  • How to tackle information overload

    In this week's Computer Weekly, we hear from Microsoft CEO Satya Nadella about how the firm wants to use cloud, machine learning and innovative datacentre technologies to open up access to information. Our new buyer's guide looks at network function virtualisation. And we examine the role of blockchain in cloud data security. Read the issue now.

    Download

  • Computer Weekly – 30 August 2016: Get protected

    It is not uncommon for computer systems to be compromised by a company's own employees. In this week's issue we look at the Sage data breach, which highlights the risk every organisation faces from its staff. Staff are often motivated by revenge, but sometimes internal logins can be compromised, enabling hackers to circumvent firewalls.

    Download

  • Hadoop Data Management and Use Cases

    In this expert e-guide, hear from pros in the analytics space, including Wayne Eckerson, explore how organizations are beginning to integrate Hadoop deeper into operational workflows, expanding the possible use cases.

    Download

  • A Computer Weekly Buyer's Guide to Hyper Converged Infrastructure

    Hyper-convergence has become a buzzword for bringing together hardware and software for multiple tasks. In this 12-page buyer's guide, Computer Weekly looks at tips for buying hyper-convergedsystems, what the technologies can do for your organisation, and how the market is shaping up.

    Download

  • Generating business value from Big Data

    For high-end, challenging applications integrating business intelligence (BI) data with other information such as Internet clickstream and social networking data, the trend is shifting toward build in the buy vs. build data warehousing debate. Access this expert resource today to learn the drivers that are creating this new trend.

    Download

  • Jargon buster guide to database management

    This Jargon Buster e-guide to contemporary database management is an aid to stepping back and thinking afresh about an area of enterprise IT that has many decades of heritage. Or, as many sometimes (arguably rudely) put it, legacy.

    Download

  • 8 ways to determine if cloud is right for your enterprise

    This e-guide recaps the key benefits of cloud storage and reveals eight points to address when getting started. Read on now to learn more.

    Download

  • What the future holds for cloud management in APAC

    In this e-guide, we dish out useful tips from APAC organisations that have successfully migrated to the cloud, the thinking behind their approaches to cloud migration and what they are doing to not only keep the lights on, but also raise their game in areas such as cloud data management and cloud security.

    Download

  • Launching a cloud-based analytics campaign: The peaks and pitfalls

    In this expert e-guide, learn about some of the major issues facing modern analytics. Read about the initial struggle of launching an operational analytics program, and how integration with cloud solutions can help alleviate these pains. In addition, read about a travel-booking company's success story of BI cloud integration.

    Download

  • A Computer Weekly buyer's guide to big data

    Companies are switching on to the advantages of big data. In this 13-pagebuyer's guide, Computer Weekly looks at data analytics, how to get value out it and the benefits of process mining.

    Download

  • Computer Weekly – 9 April 2019: Using data science at Sainsbury's

    In this week's Computer Weekly, we talk to retailer Sainsbury's about its data science strategy and how it helps the firm better understand customers. We take a detailed look at the evolution of the business intelligence market. And we find out how simulated disaster management can tackle the security skills gap. Read the issue now.

    Download

  • UK IT Priorities 2019

    The results of our 2019 IT Priorities survey are in, and they show the good news that IT budgets are rising faster than they have for many years. Cloud remains the top spending priority for this year, but IT automation is growing fast as a major investment area.

    Download

  • UK IT Priorities 2019

    The results of our 2019 IT Priorities survey are in, and they show the good news that IT budgets are rising faster than they have for many years. Cloud remains the top spending priority for this year, but IT automation is growing fast as a major investment area.

    Download

  • A Computer Weekly buyer's guide to optimising storage

    The implementation of effective storage optimisation has becomean important factor in business. In this 12-page buyer's guide,Computer Weekly looks at cost implications, the hyper-convergedinfrastructure minefield and the alternatives to shared storage

    Download

  • A Computer Weekly buyer's guide to graph databases

    In this 14-page buyer's guide, Computer Weekly looks at the technology that teases out connections between divergent datasets – and how it is revolutionising data analysis.

    Download

  • Top 10 ASEAN IT stories of 2019

    Over the year, Computer Weekly looked at how the region is embracing technology, not for its own sake but to become more nimble, secure and to overcome the challenges of a growing economy. Here are Computer Weekly's top 10 ASEAN IT stories of 2019.

    Download

  • Why the data & BI project fail rate is almost 85%

    In a recent survey, it was found that almost 60% of big data projects in the enterprise space have failed. However, in a follow up survey, it was determined that 60% is actually conservative and the fail rate is actually about 85%. But why is this rate so high? The answer lies in the shortcomings of data platforms. Read on to learn more about them.

    Download

  • Reaping the Benefits of Big Data and Real-time Analytics

    In this expert e-guide, explore strategies for embracing the big data challenge and discover how real-time analytics can help you improve business performance and leverage your high-volume data to get an edge on competitors.

    Download

  • Why Unilever combines IT with business

    In this week's Computer Weekly, we find out how consumer goods giant Unilever has changed its thinking about IT to support a global digital transformation. The UK boss of Hewlett Packard Enterprise assesses progress after the first year since HP split in two. And we examine how software development is changing in the enterprise. Read the issue now.

    Download

  • Best practices for maximizing your cloud potential

    Among challenging information technology (IT) projects, few are more intimidating than a full-fledged cloud transformation. Fortunately, with the help of some careful planning, proven best practices, and the latest tools, you can quickly realize the incredible benefits of the cloud. Read on to learn how you can make the most of your cloud journey.

    Download

  • IT Project: Business intelligence in the world of big data

    Old-school business intelligence (BI), although still important, is no longer at the cutting edge of modern IT projects. However, as the TechTarget 2016 spending survey shows, business intelligence is very much a priority for corporate IT.

    Download

  • A comparison of Azure, AWS, and Google cloud services

    AWS, Azure, and Google are all huge names in the cloud space, offering everything from big data in the cloud to serverless computing options and more. Read on for a vendor-neutral comparison of these three providers to determine which combination – if any – best fits your organization's infrastructure requirements.

    Download

  • 11 trends poised to impact data engineers in 2024

    The data space moves quickly. In this e-book, you’ll discover 11 of the most important trends that are poised to shake up the data engineering landscape this year, including data contracts, cost optimization, the evolution of modern data, and 8 more. Read on to learn how data engineering will evolve in 2024.

    Download

  • Artificial Intelligence and Machine Learning in data analytics and business applications

    Artificial Intelligence and the sub-set of it which is machine learning are hot. It's been impossible to attend an IT conference in the last year or so without encountering a dead human genius reincarnated as a software system or a robot.

    Download

  • Essential Guide: Optimising hybrid IT infrastructure

    In this essential guide, we include articles looking at best practice and key technologies in server, storage, networks, DevOps and hyperconverged systems to make the most of your hybrid IT.

    Download

  • Uber and Volkswagen: Handling ethics in IT

    Take a look at this edition of ComputerWeekly to learn more Uber, Volkswagen, and other companies that have experience with software ethics issues, how they've dealt with them, and what the consequences have been.

    Download

  • The ultimate guide to data preparation

    Poor data equates to wrong decisions, bias in the AI algorithm and flawed decision-making. What's more, data that personally identifies an individual can only be used in very specific ways, as stipulated by GDPR. Read about data preparation in this e-guide.

    Download

  • 19 machine learning interview questions and answers

    Aspiring machine learning job candidates should be fluent in varied aspects of machine learning, from statistical theory and programming concepts to general industry knowledge. Read our list of commonly asked machine learning questions and accompanying answers to help you prepare for your interview.

    Download

  • Computer Weekly – 14 May 2019: The startups transforming data analytics

    In this week's Computer Weekly, we travel to Silicon Valley to meet the emerging tech startups hoping to take data analytics to a new level. The NHS desperately needs to recruit IT professionals – we look at the job opportunities. And we examine the best ways to keep cloud costs under control. Read the issue now.

    Download

  • Top 10 ASEAN IT stories of 2020

    The booming digital economies of Southeast Asia showed much resilience in 2020, from tapping open source, data analytics and DevOps, to fending off cyber criminals who came knocking on their doors, all in the midst of an unprecedented pandemic. Here are Computer Weekly's top 10 ASEAN IT stories of 2020:

    Download

  • Event recap: VMworld 2018 Europe

    It is the 20th anniversary year for VMware and at the VMware European conference in Barcelona, the company famous for its hypervisor, showcased how it had changed the face of IT, and where it was heading next.

    Download

  • A Computer Weekly Buyer's Guide to Customer Experience

    Discover how to craft data-first CRM strategies and get tips about coordinating data sources for effective CRM, 5 ways to reduce risks for Salesforce deployments, and the pros and cons of current CRM software offerings.

    Download

  • The Future of Next-Generation SIEM

    This expert E-Guide explores what it means to be an "ideal" security information and event management (SIEM) system and takes a look into a crystal ball to explore five futures SIEMs must conquer to be considered next-gen systems. View now to learn more!

    Download

  • Software-Defined Networking Strategies for Campus WANs

    In this E-Guide from SearchNetworking.com, explore some of the issues associated with troubled OpenFlow switches, which vendors carry them, and details about a strategy that will not cause you the issues these other switches do.

    Download